search

chengzeyi / stable-fast

Best inference performance optimization framework for HuggingFace Diffusers on NVIDIA GPUs.
MIT License
1.05k stars 59 forks source link

Error importing sfast._C with PyTorch 2.1.1 with xformers >= 0.0.22 #115

Open yondonfu opened 5 months ago

yondonfu commented 5 months ago

I encountered the following error when using stable-fast to dynamically compile a diffusers image-to-image pipeline:

  File "/app/app/pipelines/image_to_image.py", line 43, in __init__
    from app.pipelines.sfast import compile_model
  File "/app/app/pipelines/sfast.py", line 1, in <module>
    from sfast.compilers.diffusion_pipeline_compiler import compile, CompilationConfig
  File "/root/.pyenv/versions/3.11.7/lib/python3.11/site-packages/sfast/__init__.py", line 23, in <module>
    import sfast._C as _C
ImportError: /root/.pyenv/versions/3.11.7/lib/python3.11/site-packages/sfast/_C.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops5zeros4callEN3c108ArrayRefINS2_6SymIntEEENS2_8optionalINS2_10ScalarTypeEEENS6_INS2_6LayoutEEEN
S6_INS2_6DeviceEEENS6_IbEE

ERROR:    Application startup failed. Exiting.

***ERROR IMPORTING sfast._C***
Unable to load stable-fast C extension.
Is it compatible with your PyTorch installation?
Or is it compatible with your CUDA version?

The compile_model() function:

from sfast.compilers.diffusion_pipeline_compiler import compile, CompilationConfig
import logging

logger = logging.getLogger(__name__)

def compile_model(model):
    config = CompilationConfig.Default()

    # xformers and Triton are suggested for achieving best performance.
    # It might be slow for Triton to generate, compile and fine-tune kernels.
    try:
        import xformers

        config.enable_xformers = True
    except ImportError:
        logger.info("xformers not installed, skip")
    # NOTE:
    # When GPU VRAM is insufficient or the architecture is too old, Triton might be slow.
    # Disable Triton if you encounter this problem.
    try:
        import triton

        config.enable_triton = True
    except ImportError:
        logger.info("Triton not installed, skip")

    model = compile(model, config)
    return model

The code that loads the diffusers pipelines and calls compile_model():

model = AutoPipelineForImage2Image.from_pretrained("stabilityai/sd-turbo", **kwargs)
model = compile_model(model)

I am running the above in Docker with:

I ran the following to install stable-fast:

pip install 'xformers>=0.0.22' 'triton>=2.1.0'
pip install https://github.com/chengzeyi/stable-fast/releases/download/v1.0.3/stable_fast-1.0.3+torch211cu121-cp311-cp311-manylinux2014_x86_64.whl

Solution

I was able to fix this error by running pip install 'xformers==0.0.23' instead of pip install 'xformers>=0.0.22'.

According to the xformers release notes, 0.0.24 pre-built wheels require PyTorch 2.2.0 so perhaps anyone using stable-fast with PyTorch < 2.2.0 needs to install xformers==0.0.23 which is the last release with pre-built wheels compatible with PyTorch < 2.2.0? Will open a PR with a small README update.

yondonfu commented 5 months ago

FWIW this was also in the release notes for xformers 0.0.24:

Triton kernels now require a GPU with compute capability 8.0 at least (A100 or newer). This is due to newer versions of triton not supporting older GPUs correctly

I don't think the above is related to the error in the OP because I was using a 3090 with compute capability 8.6:

nvidia-smi --query-gpu=compute_cap --format=csv
compute_cap
8.6
chengzeyi commented 4 months ago

If you want to use xformers with a version that the official pypi source does not provide a prebuilt binary wheels for a specific torch version, you could download the source and build yourself.

chengzeyi commented 4 months ago

And stable-fast has no binary dependencies on xformers so the failure of loading the C extension should be caused by other reasons. Anyway, xformers is just an optional requirement and you can safely ignore it.

yondonfu commented 4 months ago

@chengzeyi

Ah got it. The error loading the C extension went away after I installed xformers==0.0.23 so I assumed it was related. If it is not related, then not sure what the problem was...

Anyway, I updated the note in https://github.com/chengzeyi/stable-fast/pull/116 about the option to build the latest xformers from source for torch < 2.2.0 as well in case you think it would be helpful to have in the README - otherwise feel free to close issue/PR and I can just follow up separately if I encounter the error with loading the C extension again!

Milor123 commented 3 months ago

@chengzeyi

Ah got it. The error loading the C extension went away after I installed xformers==0.0.23 so I assumed it was related. If it is not related, then not sure what the problem was...

Anyway, I updated the note in #116 about the option to build the latest xformers from source for torch < 2.2.0 as well in case you think it would be helpful to have in the README - otherwise feel free to close issue/PR and I can just follow up separately if I encounter the error with loading the C extension again!

I've tried using xformers==0.0.23 and install de libs needs but it show me the same error, what should i do? I am in manjaro 

***ERROR IMPORTING sfast._C***
Unable to load stable-fast C extension.
Is it compatible with your PyTorch installation?
Or is it compatible with your CUDA version?

ComfyUI_stable_fast: StableFast node import failed.
Traceback (most recent call last):
  File "/home/noe/Documentos/ComfyUI/custom_nodes/ComfyUI_stable_fast/__init__.py", line 10, in <module>
    from .node import ApplyStableFastUnet
  File "/home/noe/Documentos/ComfyUI/custom_nodes/ComfyUI_stable_fast/node.py", line 2, in <module>
    from sfast.compilers.diffusion_pipeline_compiler import CompilationConfig
  File "/home/noe/Documentos/ComfyUI/venv/lib/python3.11/site-packages/sfast/__init__.py", line 23, in <module>
    import sfast._C as _C
ImportError: libcudart.so.11.0: cannot open shared object file: No such file or directory

or also 

ComfyUI_stable_fast: StableFast node import failed.
Traceback (most recent call last):
  File "/home/noe/Documentos/ComfyUI/custom_nodes/ComfyUI_stable_fast/__init__.py", line 10, in <module>
    from .node import ApplyStableFastUnet
  File "/home/noe/Documentos/ComfyUI/custom_nodes/ComfyUI_stable_fast/node.py", line 2, in <module>
    from sfast.compilers.diffusion_pipeline_compiler import CompilationConfig
  File "/home/noe/Documentos/ComfyUI/venv/lib/python3.11/site-packages/sfast/__init__.py", line 23, in <module>
    import sfast._C as _C
ImportError: /home/noe/Documentos/ComfyUI/venv/lib/python3.11/site-packages/sfast/_C.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN2at6native13scalar_tensorERKN3c106ScalarESt8optionalINS1_10ScalarTypeEES5_INS1_6LayoutEES5_INS1_6DeviceEES5_IbE
Milor123 commented 3 months ago

And stable-fast has no binary dependencies on xformers so the failure of loading the C extension should be caused by other reasons. Anyway, xformers is just an optional requirement and you can safely ignore it.

How could I solve this? I am in linux but the @yondonfu solution dont work for me.

Fengshawn commented 1 month ago

@chengzeyi Ah got it. The error loading the C extension went away after I installed xformers==0.0.23 so I assumed it was related. If it is not related, then not sure what the problem was... Anyway, I updated the note in #116 about the option to build the latest xformers from source for torch < 2.2.0 as well in case you think it would be helpful to have in the README - otherwise feel free to close issue/PR and I can just follow up separately if I encounter the error with loading the C extension again!

I've tried using xformers==0.0.23 and install de libs needs but it show me the same error, what should i do? I am in manjaro

***ERROR IMPORTING sfast._C***
Unable to load stable-fast C extension.
Is it compatible with your PyTorch installation?
Or is it compatible with your CUDA version?

ComfyUI_stable_fast: StableFast node import failed.
Traceback (most recent call last):
  File "/home/noe/Documentos/ComfyUI/custom_nodes/ComfyUI_stable_fast/__init__.py", line 10, in <module>
    from .node import ApplyStableFastUnet
  File "/home/noe/Documentos/ComfyUI/custom_nodes/ComfyUI_stable_fast/node.py", line 2, in <module>
    from sfast.compilers.diffusion_pipeline_compiler import CompilationConfig
  File "/home/noe/Documentos/ComfyUI/venv/lib/python3.11/site-packages/sfast/__init__.py", line 23, in <module>
    import sfast._C as _C
ImportError: libcudart.so.11.0: cannot open shared object file: No such file or directory

or also

ComfyUI_stable_fast: StableFast node import failed.
Traceback (most recent call last):
  File "/home/noe/Documentos/ComfyUI/custom_nodes/ComfyUI_stable_fast/__init__.py", line 10, in <module>
    from .node import ApplyStableFastUnet
  File "/home/noe/Documentos/ComfyUI/custom_nodes/ComfyUI_stable_fast/node.py", line 2, in <module>
    from sfast.compilers.diffusion_pipeline_compiler import CompilationConfig
  File "/home/noe/Documentos/ComfyUI/venv/lib/python3.11/site-packages/sfast/__init__.py", line 23, in <module>
    import sfast._C as _C
ImportError: /home/noe/Documentos/ComfyUI/venv/lib/python3.11/site-packages/sfast/_C.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN2at6native13scalar_tensorERKN3c106ScalarESt8optionalINS1_10ScalarTypeEES5_INS1_6LayoutEES5_INS1_6DeviceEES5_IbE

me too, try so many combination of different versions of pytorch, triton, xformers, encountered this two ImportError

Fengshawn commented 1 month ago

And stable-fast has no binary dependencies on xformers so the failure of loading the C extension should be caused by other reasons. Anyway, xformers is just an optional requirement and you can safely ignore it.

How could I solve this? I am in linux but the @yondonfu solution dont work for me.

I 've solved this issue by installing higher version of xformers, torch, triton