Open matbee-eth opened 1 month ago
Same here. Definitely something with the conda env. Probably need to install some cuda nvidia nvcc lib.
ImportError:
...envs/mamba/lib/python3.12/site-packages/selective_scan_cuda.cpython-312-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEi
Possible solutions: https://github.com/state-spaces/mamba/issues/169#issuecomment-2099147220
Okay so it looks like we need torch < 2.3.0 (so 2.2.2) but xformers 0.0.27 needs >=2.3.0 and mistral-inference doesn't work (for me, mamba deps of link above) with 2.3.0 but doesn't work with the xformers binaries <0.0.27. I get other errors (RuntimeError: Couldn't instantiate class <class 'mistral_inference.model.ModelArgs'> using init args dict_keys(['dim', 'n_layers', 'vocab_size']): ModelArgs.__init__() missing 5 required positional arguments: 'head_dim', 'hidden_dim', 'n_heads', 'n_kv_heads', and 'norm_eps'
) when running mistral-inference < 1.2.0. I'm trying to build xformers from source now and shot myself in the foot by not specifying MAX_JOBS with ninja (on a 12 thread CPU with 128GB, 12 jobs saturates 128GB without anything else, not even a DE, running).
Now at: AttributeError: 'Mamba2' object has no attribute 'dconv'. Did you mean: 'd_conv'?
Hmm not sure what's going on. Here a google colab showing how to install: https://colab.research.google.com/drive/1aHH4PW4eBU_R4R8pQ9BuYeOeMTiA98NF?usp=sharing
Does that work for you?
I got it to work with using conda's cuda 11.8 and python 3.9
Seems like cuda 12.1 has problems with this project
Hmm it also works for CUDA 12.1 for me. It's most likely linked to the mamba_ssm install - maybe check here: https://github.com/state-spaces/mamba?tab=readme-ov-file#installation
Working now for me too. Default cuda 12.1. Python 3.10.14 was giving me problems. This works with a fresh conda env:
conda install python==3.10.13 pip pip install mistral_inference>=1 mamba-ssm causal-conv1d
Working now for me too. Default cuda 12.1. Python 3.10.14 was giving me problems. This works with a fresh conda env:
conda install python==3.10.13 pip pip install mistral_inference>=1 mamba-ssm causal-conv1d
It does not work for me.
Working now for me too. Default cuda 12.1. Python 3.10.14 was giving me problems. This works with a fresh conda env:
conda install python==3.10.13 pip pip install mistral_inference>=1 mamba-ssm causal-conv1d
Thank you! This was my issue. There seems to be some problem with Python 3.10.14, or changing the version to 3.10.13 triggered a fix.
I'm using WSL Ubuntu 22.04 and installed CUDA with these commands (via this):
wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-keyring_1.1-1_all.debsudo
dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-5
If you have an existing cudatoolkit install, you should uninstall it before trying to install this cudatoolkit package.
You may have to add nvcc to your path afterwards (export PATH=$PATH:/usr/local/cuda/bin
). I did in my case, but I installed CUDA awhile before actually fixing the issue so it may not be necessary.
My conda env install commands:
rm -rf ~/.cache/conda ~/.cache/pip ~/.cache/huggingface # fixes the selective_scan_cuda ImportError for me
conda create -n tempenv python=3.10.13
conda activate tempenv
pip install mistral-inference>=1.2
pip install packaging # cannot install this package inline with the following pip install command
pip install mamba-ssm causal-conv1d transformers
With those commands, I end up with these versions (same as the collab notebook):
and running mamba-chat with:
mistral-chat $HOME/mistral_models/mamba-codestral-7B-v0.1 --instruct --max_tokens 256
works for me. I downloaded Codestral Mamba using the Hugging Face snapshot download Python code.
_import selective_scan_cuda ImportError: $HOME/lib/python3.10/site-packages/selective_scan_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops10zeros_like4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEENS6_INS512MemoryFormatEEE
I think it might be a version mismatch issue. I modified it to directly insert the relevant library using mamba:
_is_mamba_installed = False
from mamba_ssm.models.config_mamba import MambaConfig
from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel
# try:
# from mamba_ssm.models.config_mamba import MambaConfig
# from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel
# _is_mamba_installed = True
# except ImportError:
# _is_mamba_installed = False
and it immediately showed the error mentioned above. My environment is torch 2.1 with Python 3.10.
Python -VV
Pip Freeze
Reproduction Steps
Expected Behavior
working
Additional Context
No response
Suggested Solutions
No response