search

huggingface / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.
https://huggingface.co/docs/diffusers
Apache License 2.0
24.9k stars 5.15k forks source link

StableDiffusionXLPipeline.from_single_file Error #7389

Closed dengfenglai321 closed 3 months ago

dengfenglai321 commented 5 months ago

Describe the bug

version: 0.27.0, 0.27.1 when use StableDiffusionXLPipeline.from_single_file,error occur:

  File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py", line 383, in encode_prompt
    prompt_embeds = text_encoder(text_input_ids.to(device), output_hidden_states=True)
  File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 822, in forward
    return self.text_model(
  File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 730, in forward
    hidden_states = self.embeddings(input_ids=input_ids, position_ids=position_ids)
  File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 229, in forward
    position_embeddings = self.position_embedding(position_ids)
  File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    return F.embedding(
  File "aaa/anaconda/envs/sd-webui/lib/python3.10/site-packages/torch/nn/functional.py", line 2210, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.HalfTensor instead (while checking arguments for embedding)

but version 0.26.3 is OK

Reproduction

pipe = StableDiffusionXLPipeline.from_single_file('xxxxxxx.safetensors',
 original_config_file='aaaa/sd_xl_base.yaml', 
 torch_dtype=torch.float16, 
 local_files_only=True,
 use_safetensors=True,
 add_watermarker=False,
#  safety_checker=None,
#  requires_safety_checker=False
 )
print("22222222222222")

pipe.to('cuda')
prompt = "black denim,long jeans,1girl,datou,"
negative_prompt = '(worst quality:2.1),(low quality:2.1),(normal quality:2.1)'

# sdxl
image1 = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_inference_steps=50,
    eta=1.0,
).images[0]

image1.save("test.png")

Logs

No response

System Info

diffuser 0.27.0 0.27.1

Who can help?

No response

yiyixuxu commented 5 months ago

sorry I can't reproduce this, this runs for me

the config file is https://raw.githubusercontent.com/Stability-AI/generative-models/main/configs/inference/sd_xl_base.yaml

from diffusers import StableDiffusionXLPipeline
import torch

repo_id = "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors"
pipe = StableDiffusionXLPipeline.from_single_file(repo_id,
 original_config_file='sd_xl_base.yaml', 
 torch_dtype=torch.float16, 
 local_files_only=True,
 use_safetensors=True,
 add_watermarker=False,
 )

pipe.enable_model_cpu_offload()
prompt = "black denim,long jeans,1girl,datou,"
negative_prompt = '(worst quality:2.1),(low quality:2.1),(normal quality:2.1)'

# sdxl
image1 = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_inference_steps=50,
    eta=1.0,
).images[0]

image1.save("yiyi_test_out.png")
dengfenglai321 commented 5 months ago

sorry I can't reproduce this, this runs for me

the config file is https://raw.githubusercontent.com/Stability-AI/generative-models/main/configs/inference/sd_xl_base.yaml

from diffusers import StableDiffusionXLPipeline
import torch

repo_id = "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors"
pipe = StableDiffusionXLPipeline.from_single_file(repo_id,
 original_config_file='sd_xl_base.yaml', 
 torch_dtype=torch.float16, 
 local_files_only=True,
 use_safetensors=True,
 add_watermarker=False,
 )

pipe.enable_model_cpu_offload()
prompt = "black denim,long jeans,1girl,datou,"
negative_prompt = '(worst quality:2.1),(low quality:2.1),(normal quality:2.1)'

# sdxl
image1 = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_inference_steps=50,
    eta=1.0,
).images[0]

image1.save("yiyi_test_out.png")

what's your version? 0.26.3 is OK, but 0.27.0/0.27.1/0.27.2 occur error sd_xl_base_1.0.safetensors sd_xl_base.yaml

sayakpaul commented 5 months ago

Unable to reproduce: https://colab.research.google.com/gist/sayakpaul/62e5e57f34161122b8a1439906b84c51/scratchpad.ipynb.

cbstyle commented 4 months ago

Have you solved the problem?

github-actions[bot] commented 3 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

yiyixuxu commented 3 months ago

closing now since we are not able to reproduce on our end if this is still an issue, please let us know and we will reopen it!

eija06 commented 3 months ago

I am able to reproduce it with the following minimal code:

from diffusers import StableDiffusionXLPipeline import torch

pipeline = StableDiffusionXLPipeline.from_single_file( "/workspace/stable-diffusion-webui/models/Stable-diffusion/realismEngineSDXL_v30VAE.safetensors", torch_dtype=torch.float16 ).to("cuda")

image = pipeline(prompt="photo").images[0]

The venv is absl-py==1.4.0 accelerate==0.21.0 addict==2.4.0 aenum==3.1.15 aiofiles==23.2.1 aiohttp==3.8.5 aiosignal==1.3.1 altair==5.1.1 antlr4-python3-runtime==4.9.3 anyio==3.7.1 argon2-cffi==23.1.0 argon2-cffi-bindings==21.2.0 arrow==1.2.3 asttokens==2.2.1 async-lru==2.0.4 async-timeout==4.0.3 attrs==23.1.0 av==10.0.0 Babel==2.12.1 backcall==0.2.0 basicsr==1.4.2 beautifulsoup4==4.12.2 bleach==6.0.0 blendmodes==2022 boltons==23.0.0 cachetools==5.3.1 certifi==2023.7.22 cffi==1.15.1 chardet==5.2.0 charset-normalizer==3.2.0 clean-fid==0.1.35 click==8.1.7 clip==1.0 cmake==3.25.0 comm==0.1.4 contourpy==1.1.0 cssselect2==0.7.0 cycler==0.11.0 debugpy==1.6.7.post1 decorator==5.1.1 defusedxml==0.7.1 deprecation==2.1.0 diffusers==0.27.2 dynamicprompts==0.31.0 einops==0.4.1 exceptiongroup==1.1.3 executing==1.2.0 facexlib==0.3.0 fake-useragent==1.5.1 fastapi==0.94.0 fastjsonschema==2.18.0 ffmpy==0.3.1 filelock==3.9.0 filterpy==1.4.5 flatbuffers==23.5.26 fonttools==4.42.1 fqdn==1.5.1 frozenlist==1.4.0 fsspec==2023.6.0 ftfy==6.1.1 future==0.18.3 fvcore==0.1.5.post20221221 gdown==4.7.1 gfpgan==1.3.8 gitdb==4.0.10 GitPython==3.1.32 google-auth==2.22.0 google-auth-oauthlib==1.0.0 gradio==3.41.2 gradio_client==0.5.0 grpcio==1.57.0 h11==0.12.0 httpcore==0.15.0 httpx==0.24.1 huggingface-hub==0.23.0 idna==3.4 imageio==2.31.2 imageio-ffmpeg==0.4.8 importlib-metadata==6.8.0 importlib-resources==6.0.1 inflection==0.5.1 iopath==0.1.10 ipykernel==6.25.1 ipython==8.14.0 ipywidgets==8.1.0 isoduration==20.11.0 jedi==0.19.0 Jinja2==3.1.2 json5==0.9.14 jsonmerge==1.8.0 jsonpointer==2.4 jsonschema==4.19.0 jsonschema-specifications==2023.7.1 jupyter-events==0.7.0 jupyter-lsp==2.2.0 jupyter_client==8.3.1 jupyter_core==5.3.1 jupyter_server==2.7.3 jupyter_server_terminals==0.4.4 jupyterlab==4.0.5 jupyterlab-pygments==0.2.2 jupyterlab-widgets==3.0.8 jupyterlab_server==2.24.0 kiwisolver==1.4.5 kornia==0.6.7 lark==1.1.2 lazy_loader==0.3 lightning-utilities==0.9.0 lit==15.0.7 llvmlite==0.40.1 lmdb==1.4.1 lpips==0.1.4 lxml==4.9.3 lycoris-lora==2.2.0.post3 Markdown==3.4.4 markdown-it-py==3.0.0 MarkupSafe==2.1.3 matplotlib==3.7.2 matplotlib-inline==0.1.6 mdurl==0.1.2 mediapipe==0.10.3 mistune==3.0.1 mpmath==1.2.1 multidict==6.0.4 nbclient==0.8.0 nbconvert==7.8.0 nbformat==5.9.2 nest-asyncio==1.5.7 networkx==3.0 notebook_shim==0.2.3 numba==0.57.1 numexpr==2.8.5 numpy==1.23.5 oauthlib==3.2.2 omegaconf==2.2.3 open-clip-torch==2.20.0 opencv-contrib-python==4.8.0.76 opencv-python==4.8.0.76 orjson==3.9.5 overrides==7.4.0 packaging==23.1 pandas==2.1.0 pandocfilters==1.5.0 parso==0.8.3 pexpect==4.8.0 pickleshare==0.7.5 piexif==1.1.3 Pillow==9.5.0 PIMS==0.6.1 platformdirs==3.10.0 portalocker==2.7.0 prometheus-client==0.17.1 prompt-toolkit==3.0.39 protobuf==3.20.0 psutil==5.9.5 ptyprocess==0.7.0 pure-eval==0.2.2 py-cpuinfo==9.0.0 pyasn1==0.5.0 pyasn1-modules==0.3.0 pycparser==2.21 pydantic==1.10.12 pydub==0.25.1 Pygments==2.16.1 pyparsing==3.0.9 PySocks==1.7.1 python-dateutil==2.8.2 python-json-logger==2.0.7 python-multipart==0.0.6 pytorch-lightning==1.9.4 pytz==2023.3 PyWavelets==1.4.1 PyYAML==6.0.1 pyzmq==25.1.1 realesrgan==0.3.0 referencing==0.30.2 regex==2023.8.8 reportlab==4.0.4 requests==2.31.0 requests-oauthlib==1.3.1 resize-right==0.0.2 rfc3339-validator==0.1.4 rfc3986-validator==0.1.1 rich==13.5.2 rpds-py==0.10.0 rsa==4.9 safetensors==0.3.1 scikit-image==0.21.0 scipy==1.11.2 seaborn==0.13.2 semantic-version==2.10.0 Send2Trash==1.8.2 sentencepiece==0.1.99 six==1.16.0 slicerator==1.1.0 smmap==5.0.0 sniffio==1.3.0 sounddevice==0.4.6 soupsieve==2.4.1 stack-data==0.6.2 starlette==0.26.1 svglib==1.5.1 sympy==1.11.1 tabulate==0.9.0 tb-nightly==2.15.0a20230831 tensorboard-data-server==0.7.1 termcolor==2.3.0 terminado==0.17.1 thop==0.1.1.post2209072238 tifffile==2023.8.30 timm==0.9.2 tinycss2==1.2.1 tokenizers==0.13.3 tomesd==0.1.3 toml==0.10.2 tomli==2.0.1 toolz==0.12.0 torch==2.0.1+cu118 torchaudio==2.0.2+cu118 torchdiffeq==0.2.3 torchmetrics==1.1.1 torchsde==0.2.5 torchvision==0.15.2+cu118 tornado==6.3.3 tqdm==4.66.1 traitlets==5.9.0 trampoline==0.1.2 transformers==4.30.2 triton==2.0.0 typing_extensions==4.7.1 tzdata==2023.3 ultralytics==8.2.8 uri-template==1.3.0 urllib3==1.26.16 uvicorn==0.23.2 wcwidth==0.2.6 webcolors==1.13 webencodings==0.5.1 websocket-client==1.6.2 websockets==11.0.3 Werkzeug==2.3.7 widgetsnbextension==4.0.8 xformers==0.0.21 yacs==0.1.8 yapf==0.40.1 yarl==1.9.2 zipp==3.16.2 ZipUnicode==1.1.1

I am NOT getting this error with a fresh venv installed with the most recent versions available to pip. So you could close this bug report as fixed - but you might get this report again, because the venv posted above is in active use on many machines - it is the venv that stable diffusion A1111 webui creates.