-
# URL
- https://arxiv.org/abs/2307.09288
# Affiliations
- Hugo Touvron, N/A
- Louis Martin, N/A
- Kevin Stone, N/A
- Peter Albert, N/A
- Amjad Almahairi, N/A
- Yasmine Babaei, N/A
- Niko…
-
### 🐛 Describe the bug
when trying to train both LoRA layers on the base model and also set modules_to_save on the lora config which makes the embeddings layers trainable (my assumption is it also ap…
-
## Performance degradation in A100 GPU
- Vanilla Attention: 3.87ms
- Sparse Attention: 9.33ms
- Memory Efficient Attention: 6.34ms
- Sparse Attention is 2.4x slower than Vanilla Attention
- Memor…
-
```
Traceback (most recent call last):
File "test_mambav2.py", line 6, in
from mamba_ssm import Mamba
File "/home/test/miniconda3/envs/mamba/lib/python3.8/site-packages/mamba_ssm/__init__…
-
# ❓ Questions and Help
I'm new to xformers. I need to use Transformer Encoders to train on a dataset with a very large variation in sample lengths. My original code was:
```python
tokens = [token…
-
### Your current environment
why is it important:
This is a prerequisite to the work on enabling troch.compile on vllm, we need to be able to build vllm with nightly so that we can iterate on chan…
-
### Checklist
- [X] The issue exists after disabling all extensions
- [X] The issue exists on a clean installation of webui
- [X] The issue is caused by an extension, but I believe it is caused by a …
-
I have Anaconda installed on my windows machine, so I only followed the instructions below `Activate conda environment`. So create the new environment, activate it and install the dependencies via cop…
-
Looking for directions where to head with the following error that I'm getting all the sudden
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.1.0+cu12…
-
I am currently attempting to port a llama-like model architecture from pure pytorch to TransformerEngine's pytorch classes.
However, I have been unable to obtain identical results in certain cases.…