Open Mr-Natural opened 2 months ago
This may be unrelated, but is your torch the cpu version instead of the cuda?
Thanks for the question. The version I am using is GPU enabled one. Details: torch.version = 2.3.1+cu118
FA2 does not support cuda 11.8 on windows. You'd need to update to cuda 12 and install a prebuilt wheel from my fork https://github.com/bdashore3/flash-attention/releases
Thank you for taking the time to respond. I'll follow your guidance and all going to plan things will be operational soon!
FA2 does not support cuda 11.8 on windows. You'd need to update to cuda 12 and install a prebuilt wheel from my fork https://github.com/bdashore3/flash-attention/releases
I installed CUDA 12.2 and installed the fitting wheel from your repo. However, just running
import flash_attn
print(flash_attn.__version__)
I get the error that flash_attn_2_cuda module cannot be found
Traceback (most recent call last):
File "C:\Users\tuete\PycharmProjects\Florence-2-large-ft\testing.py", line 1, in <module>
import flash_attn
File "C:\Dev\Python3.10\lib\site-packages\flash_attn\__init__.py", line 3, in <module>
from flash_attn.flash_attn_interface import (
File "C:\Dev\Python3.10\lib\site-packages\flash_attn\flash_attn_interface.py", line 10, in <module>
import flash_attn_2_cuda as flash_attn_cuda
ImportError: DLL load failed while importing flash_attn_2_cuda: Das angegebene Modul wurde nicht gefunden.
Any idea how to fix that?
Nevermind, it was because I had the wrong torch version (without CUDA available), it worked after uninstalling torch and reinstalling with the command from https://pytorch.org/get-started/locally/
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
Thanks for the reply @zymox
I have cuda 12.5 installed. Is this version compatible? ... Update: it worked
I have cuda 12.5 installed. Is this version compatible? ... Update: it worked
Hey! How did you get it to work? I have cuda 12.5 too.
I have cuda 12.5 installed. Is this version compatible? ... Update: it worked
Hey! How did you get it to work? I have cuda 12.5 too.
I downloaded and ran the wheel correct wheel: https://github.com/bdashore3/flash-attention/releases
Also, make sure you have the right version of cuda installed. Here is the latest version for windows:
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
If you need another version, you can find it here: https://pytorch.org/get-started/locally/
So basically what @zymox said.
@KOG-Nisse Thank you! That solved my problem. I think we can close this issue.
FA2 does not support cuda 11.8 on windows. You'd need to update to cuda 12 and install a prebuilt wheel from my fork https://github.com/bdashore3/flash-attention/releases
HI bro, how can i run this wheel. My cuda is 12.1 pytorch 2.3.1
First, download a correct wheel from the link you mentioned. If you don't know how to install it, follow instructions in this link: in stackoverflow
First, download a correct wheel from the link you mentioned. If you don't know how to install it, follow instructions in this link: in stackoverflow
I check into your wheel link and see the package for cuda 12.3, but my cuda is 12.1 , is it compatible?
Yes
I am encountering an error while attempting to install the flash-attn library on my Windows 11 machine with CUDA 11.8. Despite having the nvcc compiler and CUDA properly installed and accessible, the installation fails due to a supposed missing CUDA_HOME environment variable.
Setup Information: Operating System: Windows 11 (version 10.0.22631.3593) CUDA Version: 11.8 PyTorch Version: 2.3.0+cpu NVIDIA Driver Version: 522.06
Steps to Reproduce:
Set CUDA_HOME environment variable: setx CUDA_HOME "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8" setx PATH "%CUDA_HOME%\bin;%PATH%" setx PATH "%CUDA_HOME%\lib64;%PATH%"
Verify CUDA_HOME and nvcc echo %CUDA_HOME% nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Sep_21_10:41:10_Pacific_Daylight_Time_2022 Cuda compilation tools, release 11.8, V11.8.89 Build cuda_11.8.r11.8/compiler.31833905_0
Create and activate a new Conda environment: conda create -n phivision_clean python=3.9 conda activate phivision_clean
Install prerequisites: pip install packaging ninja pip cache purge set MAX_JOBS=4
Attempt to install FlashAttention pip install flash-attn --no-build-isolation
Error Log: Collecting flash-attn Downloading flash_attn-2.5.9.post1.tar.gz (2.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 5.7 MB/s eta 0:00:00 Preparing metadata (setup.py) ... error error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [22 lines of output] fatal: not a git repository (or any of the parent directories): .git C:\Users\allan\AppData\Local\Temp\pip-install-9pn3ykxp\flash-attn_1cfbe659b3424b648b092c7a32f1614f\setup.py:78: UserWarning: flash_attn was requested, but nvcc was not found. Are you sure your environment has nvcc available? If you're installing within a container from https://hub.docker.com/r/pytorch/pytorch, only images whose names contain 'devel' will provide nvcc. warnings.warn( Traceback (most recent call last): File "", line 2, in
File "", line 34, in
File "C:\Users\allan\AppData\Local\Temp\pip-install-9pn3ykxp\flash-attn_1cfbe659b3424b648b092c7a32f1614f\setup.py", line 134, in
CUDAExtension(
File "C:\Users\allan\miniconda3\envs\phivision\Lib\site-packages\torch\utils\cpp_extension.py", line 1077, in CUDAExtension
library_dirs += library_paths(cuda=True)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\allan\miniconda3\envs\phivision\Lib\site-packages\torch\utils\cpp_extension.py", line 1211, in library_paths
paths.append(_join_cuda_home(lib_dir))
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\allan\miniconda3\envs\phivision\Lib\site-packages\torch\utils\cpp_extension.py", line 2419, in _join_cuda_home
raise OSError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed
× Encountered error while generating package metadata. ╰─> See above for output.
note: This is an issue with the package mentioned above, not pip. hint: See above for details.
Additional Information:
nvcc output: nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Sep_21_10:41:10_Pacific_Daylight_Time_2022 Cuda compilation tools, release 11.8, V11.8.89 Build cuda_11.8.r11.8/compiler.31833905_0
nvidia-smi output: Sat Jun 8 21:58:06 2024 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 522.06 Driver Version: 522.06 CUDA Version: 11.8 | |-------------------------------+----------------------+----------------------+ | GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... WDDM | 00000000:01:00.0 On | N/A | | 31% 35C P8 13W / 175W | 1125MiB / 8192MiB | 2% Default | | | | N/A | +-------------------------------+----------------------+----------------------+
Notes: Despite setting CUDA_HOME and confirming nvcc is available, the installation script does not seem to recognize the CUDA_HOME variable. The error might also be linked to residual files or incorrect environment setup from previous installation attempts. Thank you for your assistance.