Error Installing FlashAttention on Windows 11 with CUDA 11.8 - "CUDA_HOME environment variable is not set"

Mr-Natural commented 2 months ago

I am encountering an error while attempting to install the flash-attn library on my Windows 11 machine with CUDA 11.8. Despite having the nvcc compiler and CUDA properly installed and accessible, the installation fails due to a supposed missing CUDA_HOME environment variable.

Setup Information: Operating System: Windows 11 (version 10.0.22631.3593) CUDA Version: 11.8 PyTorch Version: 2.3.0+cpu NVIDIA Driver Version: 522.06

Steps to Reproduce:

Set CUDA_HOME environment variable: setx CUDA_HOME "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8" setx PATH "%CUDA_HOME%\bin;%PATH%" setx PATH "%CUDA_HOME%\lib64;%PATH%"
Verify CUDA_HOME and nvcc echo %CUDA_HOME% nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Sep_21_10:41:10_Pacific_Daylight_Time_2022 Cuda compilation tools, release 11.8, V11.8.89 Build cuda_11.8.r11.8/compiler.31833905_0

Create and activate a new Conda environment: conda create -n phivision_clean python=3.9 conda activate phivision_clean
Install prerequisites: pip install packaging ninja pip cache purge set MAX_JOBS=4
Attempt to install FlashAttention pip install flash-attn --no-build-isolation

Error Log: Collecting flash-attn Downloading flash_attn-2.5.9.post1.tar.gz (2.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 5.7 MB/s eta 0:00:00 Preparing metadata (setup.py) ... error error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [22 lines of output] fatal: not a git repository (or any of the parent directories): .git C:\Users\allan\AppData\Local\Temp\pip-install-9pn3ykxp\flash-attn_1cfbe659b3424b648b092c7a32f1614f\setup.py:78: UserWarning: flash_attn was requested, but nvcc was not found. Are you sure your environment has nvcc available? If you're installing within a container from https://hub.docker.com/r/pytorch/pytorch, only images whose names contain 'devel' will provide nvcc. warnings.warn( Traceback (most recent call last): File "", line 2, in File "", line 34, in File "C:\Users\allan\AppData\Local\Temp\pip-install-9pn3ykxp\flash-attn_1cfbe659b3424b648b092c7a32f1614f\setup.py", line 134, in CUDAExtension( File "C:\Users\allan\miniconda3\envs\phivision\Lib\site-packages\torch\utils\cpp_extension.py", line 1077, in CUDAExtension library_dirs += library_paths(cuda=True) ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\allan\miniconda3\envs\phivision\Lib\site-packages\torch\utils\cpp_extension.py", line 1211, in library_paths paths.append(_join_cuda_home(lib_dir)) ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\allan\miniconda3\envs\phivision\Lib\site-packages\torch\utils\cpp_extension.py", line 2419, in _join_cuda_home raise OSError('CUDA_HOME environment variable is not set. ' OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.

  torch.__version__  = 2.3.0+cpu

  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed

× Encountered error while generating package metadata. ╰─> See above for output.

note: This is an issue with the package mentioned above, not pip. hint: See above for details.

Additional Information:

nvcc output: nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Sep_21_10:41:10_Pacific_Daylight_Time_2022 Cuda compilation tools, release 11.8, V11.8.89 Build cuda_11.8.r11.8/compiler.31833905_0

Notes: Despite setting CUDA_HOME and confirming nvcc is available, the installation script does not seem to recognize the CUDA_HOME variable. The error might also be linked to residual files or incorrect environment setup from previous installation attempts. Thank you for your assistance.

Tellterubouzu commented 2 months ago

This may be unrelated, but is your torch the cpu version instead of the cuda?

Mr-Natural commented 2 months ago

Thanks for the question. The version I am using is GPU enabled one. Details: torch.version = 2.3.1+cu118

bdashore3 commented 2 months ago

FA2 does not support cuda 11.8 on windows. You'd need to update to cuda 12 and install a prebuilt wheel from my fork https://github.com/bdashore3/flash-attention/releases

Mr-Natural commented 2 months ago

Thank you for taking the time to respond. I'll follow your guidance and all going to plan things will be operational soon!

zymox commented 2 months ago

FA2 does not support cuda 11.8 on windows. You'd need to update to cuda 12 and install a prebuilt wheel from my fork https://github.com/bdashore3/flash-attention/releases

I installed CUDA 12.2 and installed the fitting wheel from your repo. However, just running

import flash_attn
print(flash_attn.__version__)

I get the error that flash_attn_2_cuda module cannot be found

Traceback (most recent call last):
  File "C:\Users\tuete\PycharmProjects\Florence-2-large-ft\testing.py", line 1, in <module>
    import flash_attn
  File "C:\Dev\Python3.10\lib\site-packages\flash_attn\__init__.py", line 3, in <module>
    from flash_attn.flash_attn_interface import (
  File "C:\Dev\Python3.10\lib\site-packages\flash_attn\flash_attn_interface.py", line 10, in <module>
    import flash_attn_2_cuda as flash_attn_cuda
ImportError: DLL load failed while importing flash_attn_2_cuda: Das angegebene Modul wurde nicht gefunden.

Any idea how to fix that?

zymox commented 2 months ago

Nevermind, it was because I had the wrong torch version (without CUDA available), it worked after uninstalling torch and reinstalling with the command from https://pytorch.org/get-started/locally/

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

Mr-Natural commented 1 month ago

Thanks for the reply @zymox

KOG-Nisse commented 1 month ago

I have cuda 12.5 installed. Is this version compatible? ... Update: it worked

Swinster6 commented 1 month ago

I have cuda 12.5 installed. Is this version compatible? ... Update: it worked

Hey! How did you get it to work? I have cuda 12.5 too.

KOG-Nisse commented 1 month ago

I have cuda 12.5 installed. Is this version compatible? ... Update: it worked

Hey! How did you get it to work? I have cuda 12.5 too.

I downloaded and ran the wheel correct wheel: https://github.com/bdashore3/flash-attention/releases

Also, make sure you have the right version of cuda installed. Here is the latest version for windows:

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

If you need another version, you can find it here: https://pytorch.org/get-started/locally/

So basically what @zymox said.

Rahman2001 commented 2 weeks ago

@KOG-Nisse Thank you! That solved my problem. I think we can close this issue.

ucaokylong commented 3 days ago

FA2 does not support cuda 11.8 on windows. You'd need to update to cuda 12 and install a prebuilt wheel from my fork https://github.com/bdashore3/flash-attention/releases

HI bro, how can i run this wheel. My cuda is 12.1 pytorch 2.3.1

Rahman2001 commented 3 days ago

First, download a correct wheel from the link you mentioned. If you don't know how to install it, follow instructions in this link: in stackoverflow

ucaokylong commented 1 day ago

First, download a correct wheel from the link you mentioned. If you don't know how to install it, follow instructions in this link: in stackoverflow

I check into your wheel link and see the package for cuda 12.3, but my cuda is 12.1 , is it compatible?

tridao commented 1 day ago

Yes

Dao-AILab / flash-attention

Error Installing FlashAttention on Windows 11 with CUDA 11.8 - "CUDA_HOME environment variable is not set" #982