search

facebookresearch / xformers

Hackable and optimized Transformers building blocks, supporting a composable construction.
https://facebookresearch.github.io/xformers/
Other
8.61k stars 611 forks source link

Expected release date for a version that supports Torch 2.1.0? #870

Closed caffeinism closed 1 year ago

caffeinism commented 1 year ago

❓ Questions and Help

Thanks for the great project. Recently torch 2.1.0 was released as stable. Are there any build plans or development releases for this?

patrickvonplaten commented 1 year ago

Also very much needed for Diffusers: https://github.com/huggingface/diffusers

patrickvonplaten commented 1 year ago

cc @DN6

cocktailpeanut commented 1 year ago

This is breaking a lot of apps that used to work before even though nothing has changed.

For example, any project that requires installation of torch for cuda11.8 will fail to install because of the torch version conflict:

(env) C:\Users\Shadow\pinokio\api\illusion.pinokio.git>pip install xformers
Collecting xformers
  Using cached xformers-0.0.22-cp310-cp310-win_amd64.whl (97.6 MB)
Collecting torch==2.0.1
  Using cached torch-2.0.1-cp310-cp310-win_amd64.whl (172.3 MB)
Requirement already satisfied: numpy in c:\users\shadow\pinokio\api\illusion.pinokio.git\env\lib\site-packages (from xformers) (1.24.1)
Requirement already satisfied: sympy in c:\users\shadow\pinokio\api\illusion.pinokio.git\env\lib\site-packages (from torch==2.0.1->xformers) (1.12)
Requirement already satisfied: networkx in c:\users\shadow\pinokio\api\illusion.pinokio.git\env\lib\site-packages (from torch==2.0.1->xformers) (3.0)
Requirement already satisfied: typing-extensions in c:\users\shadow\pinokio\api\illusion.pinokio.git\env\lib\site-packages (from torch==2.0.1->xformers) (4.4.0)
Requirement already satisfied: filelock in c:\users\shadow\pinokio\api\illusion.pinokio.git\env\lib\site-packages (from torch==2.0.1->xformers) (3.9.0)
Requirement already satisfied: jinja2 in c:\users\shadow\pinokio\api\illusion.pinokio.git\env\lib\site-packages (from torch==2.0.1->xformers) (3.1.2)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\shadow\pinokio\api\illusion.pinokio.git\env\lib\site-packages (from jinja2->torch==2.0.1->xformers) (2.1.2)
Requirement already satisfied: mpmath>=0.19 in c:\users\shadow\pinokio\api\illusion.pinokio.git\env\lib\site-packages (from sympy->torch==2.0.1->xformers) (1.3.0)
Installing collected packages: torch, xformers
  Attempting uninstall: torch
    Found existing installation: torch 2.1.0+cu118
    Uninstalling torch-2.1.0+cu118:
      Successfully uninstalled torch-2.1.0+cu118
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.16.0+cu118 requires torch==2.1.0+cu118, but you have torch 2.0.1 which is incompatible.
torchaudio 2.1.0+cu118 requires torch==2.1.0+cu118, but you have torch 2.0.1 which is incompatible.
schmidt-ai commented 1 year ago

+1

moinnadeem commented 1 year ago

+1

gugarosa commented 1 year ago

@cocktailpeanut Not that this is a permanent fix, but to avoid clashing with my torch's version, I leave xformers installation as the last one and use: pip install --no-dependencies xformers.

Of course it may say there is a mismatch between required and installed torch versions, but at least it will not overwrite it 😄

cocktailpeanut commented 1 year ago

@gugarosa yes that's one way around this, but I've written too many installation scripts that are all suddenly broken because of this, and don't want to go back and update all of them, just to see the xformers team make the update soon afterwards. Also in my case it's not that simple because these installation scripts are not manually calling pip install xformers, but xformers happens to be just one of many dependencies included in requirements.txt.

Really hope this would be updated soon. Otherwise I will have to go back and update all my install scripts to add an ad hoc solution to each.

But even if the team is not going to implement this any time soon, would appreciate any kind of ETA from the team, so I can just give up and do what I mentioned above for the time being.

danthe3rd commented 1 year ago

Hi, Thanks for raising this issue. ETA is maximum 1 week - most likely tomorrow if it's just a matter of changing the version of pytorch we build for. I'll keep you posted, and that's definitively on us, we should have anticipated that change

schmidt-ai commented 1 year ago

No problem. Thanks for the quick turnaround!

danthe3rd commented 1 year ago

We should be able to get binaries on pypi/conda for pt2.1 today (cuda 11.8). We will add support for cuda 12 via conda only next week.

daxijiu commented 1 year ago

We should be able to get binaries on pypi/conda for pt2.1 today (cuda 11.8). We will add support for cuda 12 via conda only next week.

When will the cuda12 ver. binaries on pypi(windows)?

danthe3rd commented 1 year ago

We just landed d20354a68acf6ce4d82ccda51efd10f8385cb64a which should create builds on conda/pip (in ~3-4 hours to give CI time to run). These will be dev versions for now (so installable with pip install --pre xformers), let me know if it's easier if we tag a proper release.

When will the cuda12 ver. binaries on pypi(windows)?

This is not planned yet. Maybe in a month or so. Pypi makes it difficult to have at the same time cuda 11.8 AND cuda 12.X builds

daxijiu commented 1 year ago

We just landed d20354a which should create builds on conda/pip (in ~3-4 hours to give CI time to run). These will be dev versions for now (so installable with pip install --pre xformers), let me know if it's easier if we tag a proper release.

When will the cuda12 ver. binaries on pypi(windows)?

This is not planned yet. Maybe in a month or so. Pypi makes it difficult to have at the same time cuda 11.8 AND cuda 12.X builds

Perhaps you can host the binary files of the CUDA 12 version on your own mirror server (similar to PyTorch), so that users can install it using a command like "pip install xformers --index-url https://xxxxxx.xformers.xxxx/whl/cu121"

danthe3rd commented 1 year ago

@daxijiu yes we would like to do something like that, but there is some infra to setup on our side to get there

schmidt-ai commented 1 year ago

Upvote for CUDA 12.1 binaries!

cocktailpeanut commented 1 year ago

@danthe3rd thank you for the update!

These will be dev versions for now (so installable with pip install --pre xformers), let me know if it's easier if we tag a proper release.

I actually think it will be much easier if it was a proper release. Just to give you an example, most projects have xformers included in the requirements.txt file and the dependencies are installed using pip install -r requirements.txt.

This means, if a project already had a xformers dependency, they all need to go back and update their requirements.txt file, which means for a vast majority of projects using xformers, the project maintainers need to manually make changes to get this to work if this is tagged as --pre.

The problem is, you guys will at some point roll out a proper release, and all those project maintainers will again have to go back and update their requirements.txt files, and those maintainers also know that this will happen, so many of them simply won't do it and just keep them broken and wait for you to release the proper release.

This means many projects will stay broken until an official version is released. So would appreciate if you release an official version. That way all the projects with vanilla xformers dependency included in the requirements.txt file will "just work" automatically without anyone doing anything. So would appreciate it if you could make a proper release if it isn't too much work....Thank you!

NeedsMoar commented 1 year ago

I was able to build using the listed command on the main page against CUDA 12.1 on Windows 10 Pro Workstation, it spewed a bunch of warnings regarding CL rewriting some command line options... Possibly VS 2022 preview stuff still took under 2 minutes. The build still completed successfully and the xformers.info command showed good versions of everything. I'm running that with the Pytorch 2.10+CUDA12.1 prebuilt wheel. It's roughly the same speed as the previous version in ComfyUI inference but it at least allowed getting everything reasonably up to date without issues. As long as nothing tries to be fancy and install some outdated version it'll do until there's a release.

freewill777 commented 1 year ago

waiting for this

thiagojramos commented 1 year ago

I was able to build using the listed command on the main page against CUDA 12.1 on Windows 10 Pro Workstation, it spewed a bunch of warnings regarding CL rewriting some command line options... Possibly VS 2022 preview stuff still took under 2 minutes. The build still completed successfully and the xformers.info command showed good versions of everything. I'm running that with the Pytorch 2.10+CUDA12.1 prebuilt wheel. It's roughly the same speed as the previous version in ComfyUI inference but it at least allowed getting everything reasonably up to date without issues. As long as nothing tries to be fancy and install some outdated version it'll do until there's a release.

How did you do that?

NeedsMoar commented 1 year ago

I was able to build using the listed command on the main page against CUDA 12.1 on Windows 10 Pro Workstation, it spewed a bunch of warnings regarding CL rewriting some command line options... Possibly VS 2022 preview stuff still took under 2 minutes. The build still completed successfully and the xformers.info command showed good versions of everything. I'm running that with the Pytorch 2.10+CUDA12.1 prebuilt wheel. It's roughly the same speed as the previous version in ComfyUI inference but it at least allowed getting everything reasonably up to date without issues. As long as nothing tries to be fancy and install some outdated version it'll do until there's a release.

How did you do that?

I just ran 

pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers

As on the main page.

Results from check (no triton, but that didn't exist before either).

C:\Programs\ComfyUI>python -m xformers.info
A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
xFormers 0.0.23+d20354a.d20231007
memory_efficient_attention.cutlassF:               available
memory_efficient_attention.cutlassB:               available
memory_efficient_attention.decoderF:               available
memory_efficient_attention.flshattF@0.0.0:         unavailable
memory_efficient_attention.flshattB@0.0.0:         unavailable
memory_efficient_attention.smallkF:                available
memory_efficient_attention.smallkB:                available
memory_efficient_attention.tritonflashattF:        unavailable
memory_efficient_attention.tritonflashattB:        unavailable
memory_efficient_attention.triton_splitKF:         unavailable
indexing.scaled_index_addF:                        available
indexing.scaled_index_addB:                        available
indexing.index_select:                             available
swiglu.dual_gemm_silu:                             available
swiglu.gemm_fused_operand_sum:                     available
swiglu.fused.p.cpp:                                available
is_triton_available:                               False
pytorch.version:                                   2.1.0+cu121
pytorch.cuda:                                      available
gpu.compute_capability:                            8.9
gpu.name:                                          NVIDIA GeForce RTX 4090
build.info:                                        available
build.cuda_version:                                1202
build.python_version:                              3.10.11
build.torch_version:                               2.1.0+cu121
build.env.TORCH_CUDA_ARCH_LIST:                    None
build.env.XFORMERS_BUILD_TYPE:                     None
build.env.XFORMERS_ENABLE_DEBUG_ASSERTIONS:        None
build.env.NVCC_FLAGS:                              None
build.env.XFORMERS_PACKAGE_FROM:                   None
build.nvcc_version:                                12.2.140
source.privacy:                                    open source

Visual Studio install is 2022, v17.8.0-pre.2.0, CL.exe is 19.38.33030, python is their official installer from the website. C:\Programs\VS2022>python -VV Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]

CUDA tools are 

C:\Programs\VS2022>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Aug_15_22:09:35_Pacific_Daylight_Time_2023
Cuda compilation tools, release 12.2, V12.2.140
Build cuda_12.2.r12.2/compiler.33191640_0

I also have a copy of 11.8 installed from before. but it didn't need to look there for anything.

Wasn't in a venv or anything since this is all I use python for. Wasn't in the VS2022 dev command prompt, NVCC handles locating the compiler. I plan on attempting to get python 3.11 setup with all of this stuff next since I'm only running 3.10 out of sheer lazy.

Didn't do anything special. Just followed the instructions, although I'd read this thread first and questioned whether it would actually work.

nirvanaws1980 commented 1 year ago

i installed torch2.1.0+cu121, however, xformers build failed on h100 at one step

pip list | grep torch torch 2.1.0+cu121 torchaudio 2.1.0+cu121 torchvision 0.16.0+cu121

build step: /usr/local/cuda/bin/nvcc -I/workspace/xformers/third_party/flash-attention/csrc/flash_attn -I/workspace/xformers/third_party/flash-attention/csrc/flash_attn/src -I/workspace/xformers/third_party/flash-attention/csrc/cutlass/include -I/usr/local/lib/python3.8/dist-packages/torch/include -I/usr/local/lib/python3.8/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/lib/python3.8/dist-packages/torch/include/TH -I/usr/local/lib/python3.8/dist-packages/torch/include/THC -I/usr/local/cuda/include -I/usr/include/python3.8 -c -c /workspace/xformers/third_party/flash-attention/csrc/flash_attn/src/flash_fwd_split_hdim64_fp16_sm80.cu -o /workspace/xformers/build/temp.linux-x86_64-cpython-38/workspace/xformers/third_party/flash-attention/csrc/flash_attn/src/flash_fwd_split_hdim64_fp16_sm80.o -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DHAS_PYTORCH --use_fast_math -UCUDA_NO_HALF_OPERATORS -UCUDA_NO_HALF_CONVERSIONS --extended-lambda -D_ENABLE_EXTENDED_ALIGNED_STORAGE -std=c++17 --generate-line-info -DNDEBUG --threads 4 --ptxas-options=-v -O3 -std=c++17 -UCUDA_NO_HALF_OPERATORS -UCUDA_NO_HALF_CONVERSIONS -UCUDA_NO_HALF2_OPERATORS -UCUDA_NO_BFLOAT16_CONVERSIONS --expt-relaxed-constexpr --expt-extended-lambda --use_fast_math --ptxas-options=-v -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_90,code=sm_90 --generate-line-info -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C_flashattention -D_GLIBCXX_USE_CXX11_ABI=0

failure: /bin/bash: line 1: 21863 Segmentation fault (core dumped) ptxas -arch=sm_90 -m64 -v -v --generate-line-info "/tmp/tmpxft_0000550b_00000000-6_flash_fwd_split_hdim64_fp16_sm80.compute_90.ptx" -o "/tmp/tmpxft_0000550b_00000000-14_flash_fwd_split_hdim64_fp16_sm80.compute_90.cubin" > /tmp/tmpxft_0000550b_00000000-16_200abc0_stdout 2> /tmp/tmpxft_0000550b_00000000-16_200abc0_stderr

ashleykleynhans commented 1 year ago

@cocktailpeanut Not that this is a permanent fix, but to avoid clashing with my torch's version, I leave xformers installation as the last one and use: pip install --no-dependencies xformers.

This did not work for me, A1111 broke in a very dramatic fashion.

moinnadeem commented 1 year ago

Hi all, if it helps anyone else, here's a copy of a xformers built for CUDA 12.1 and PyTorch 2.1.0.

https://drive.google.com/file/d/1aEERyW2-neIBS6PMG6_2J5V6K29BYd1W/view?usp=sharing

cocktailpeanut commented 1 year ago

@danthe3rd Hi any timeline for a proper release? Any kind of update would be appreciated, it's ok if you can't. It would just be great if you can let us know so we can decide whether to wait for the proper version release (so we can just install with pip install xformers without the --pre) or to go back and monkey patch our install scripts so they work for now. Thank you.

danthe3rd commented 1 year ago

We will try to get CUDA 12 builds by end of month We can make a proper release by end of week hopefully

cocktailpeanut commented 1 year ago

We can make a proper release by end of week hopefully

@danthe3rd thank you so much! Really the cuda 12 is a "nice to have" feature so not as urgent, whereas the proper release is badly needed (most projects that make use of xformers are breaking unless the project maintainers temporarily updated their dependencies to use the --pre version, for example you can see some people experiencing issues with automatic1111 stable diffusion web ui https://github.com/facebookresearch/xformers/issues/870#issuecomment-1754055285 but there are more since a lot of useful projects use xformers now)

I know i've bugged you too much on this thread so just wanted to explain the rationale. Thank you again, really appreciate it!

danthe3rd commented 1 year ago

I'm also worried with the CUDA compatibility, because by default pip install torch now installs the cuda 12 version of pytorch, while pip install (--pre) xformers installs xFormers for cuda 11.8. Not sure if both play well together, would need to do some more testing

danthe3rd commented 1 year ago

We should have CUDA 11/CUDA 12 builds hopefully soon. We should also have Flash-Attention support on windows for cuda 12 builds. I'll merge https://github.com/facebookresearch/xformers/pull/882 once the builds are ready (should be 0.0.22.post2 - to have a tagged version :) )

suede299 commented 1 year ago

0.0.22.post3 installation "No module named 'torch'" image image

SinclairCoder commented 1 year ago

This is breaking a lot of apps that used to work before even though nothing has changed.

For example, any project that requires installation of torch for cuda11.8 will fail to install because of the torch version conflict:

(env) C:\Users\Shadow\pinokio\api\illusion.pinokio.git>pip install xformers
Collecting xformers
  Using cached xformers-0.0.22-cp310-cp310-win_amd64.whl (97.6 MB)
Collecting torch==2.0.1
  Using cached torch-2.0.1-cp310-cp310-win_amd64.whl (172.3 MB)
Requirement already satisfied: numpy in c:\users\shadow\pinokio\api\illusion.pinokio.git\env\lib\site-packages (from xformers) (1.24.1)
Requirement already satisfied: sympy in c:\users\shadow\pinokio\api\illusion.pinokio.git\env\lib\site-packages (from torch==2.0.1->xformers) (1.12)
Requirement already satisfied: networkx in c:\users\shadow\pinokio\api\illusion.pinokio.git\env\lib\site-packages (from torch==2.0.1->xformers) (3.0)
Requirement already satisfied: typing-extensions in c:\users\shadow\pinokio\api\illusion.pinokio.git\env\lib\site-packages (from torch==2.0.1->xformers) (4.4.0)
Requirement already satisfied: filelock in c:\users\shadow\pinokio\api\illusion.pinokio.git\env\lib\site-packages (from torch==2.0.1->xformers) (3.9.0)
Requirement already satisfied: jinja2 in c:\users\shadow\pinokio\api\illusion.pinokio.git\env\lib\site-packages (from torch==2.0.1->xformers) (3.1.2)
Requirement already satisfied: MarkupSafe>=2.0 in c:\users\shadow\pinokio\api\illusion.pinokio.git\env\lib\site-packages (from jinja2->torch==2.0.1->xformers) (2.1.2)
Requirement already satisfied: mpmath>=0.19 in c:\users\shadow\pinokio\api\illusion.pinokio.git\env\lib\site-packages (from sympy->torch==2.0.1->xformers) (1.3.0)
Installing collected packages: torch, xformers
  Attempting uninstall: torch
    Found existing installation: torch 2.1.0+cu118
    Uninstalling torch-2.1.0+cu118:
      Successfully uninstalled torch-2.1.0+cu118
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.16.0+cu118 requires torch==2.1.0+cu118, but you have torch 2.0.1 which is incompatible.
torchaudio 2.1.0+cu118 requires torch==2.1.0+cu118, but you have torch 2.0.1 which is incompatible.

Same problem.

danthe3rd commented 1 year ago

@SinclairCoder can you post the error log you have? You should use pytorch 2.1.0 now, and the latest xformers release require 2.1.0

SinclairCoder commented 1 year ago

Thanks for your reply. I first installed torch 2.01 with cuda11.8 version. Then I installed xformers building from source pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers.

Amazingly, it upgraded my torch with the cuda12.2 version.

So amazing. I need cuda11.8. Currently, I am working on tinyllama, preparing for environment (refer to https://github.com/jzhang38/TinyLlama/blob/main/PRETRAIN.md).

danthe3rd commented 1 year ago

@SinclairCoder Can you open a new issue with the full output of pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers? Also specify if you are running on windows or linux. You might also try to install it like this:

pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers --index-url https://download.pytorch.org/whl/cu118

I'm closing this issue, as we releases 0.0.22.post4 which is built for pytorch 2.1.0 in the meantime

SinclairCoder commented 1 year ago

@SinclairCoder Can you open a new issue with the full output of pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers? Also specify if you are running on windows or linux. You might also try to install it like this:

pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers --index-url https://download.pytorch.org/whl/cu118

I'm closing this issue, as we releases 0.0.22.post4 which is built for pytorch 2.1.0 in the meantime

@danthe3rd So, I should run pip install xformers==0.0.22.post4, right? Suppose I have installed the pytorch 2.1cu118

hollowstrawberry commented 1 year ago

xformers==0.0.22.post4 depends on cu121 but Google Colab is running cu118. Any tips?

cocktailpeanut commented 1 year ago

xformers==0.0.22.post4 depends on cu121 but Google Colab is running cu118. Any tips?

Facing the same issue. Not just for google colab but some other packages.

Does this mean xformers now requires cuda 12.1 to run (and everything below it fails)?

MqLeet commented 4 months ago

Now xformers==0.0.22.post7 is compatible with torch==2.1.0+cu118.

thiagojramos commented 4 months ago

Now xformers==0.0.22.post7 is compatible with torch==2.1.0+cu118.

But the new one it's not with torch 2.3.1+cu124 or cu121

MqLeet commented 4 months ago

Now xformers==0.0.22.post7 is compatible with torch==2.1.0+cu118.

But the new one it's not with torch 2.3.1+cu124 or cu121

Yes, but I saw a project that requires torch==2.1.0 yesterday, so I tried many versions of xformers to match the torch==2.1.0 ....