search

invoke-ai / InvokeAI

InvokeAI is a leading creative engine for Stable Diffusion models, empowering professionals, artists, and enthusiasts to generate and create visual media using the latest AI-driven technologies. The solution offers an industry leading WebUI, supports terminal use through a CLI, and serves as the foundation for multiple commercial products.
https://invoke-ai.github.io/InvokeAI/
Apache License 2.0
22.38k stars 2.32k forks source link

[bug]: v4.0.0 inpainting RuntimeError: expected scalar type float but found c10::Half #6022

Open pravbk100 opened 4 months ago

pravbk100 commented 4 months ago

Is there an existing issue for this problem?

Operating system

macOS

GPU vendor

Apple Silicon (MPS)

GPU model

M1 mac mini

GPU VRAM

16gb

Version number

4.0.0

Browser

safari 17.1.2

Python dependencies

No response

What happened

I am getting

RuntimeError: expected scalar type float but found c10::Half

whenever i try to use unified canvas inpainting or outpainting. It works with SD 1.5 inpainting model but throws this error if i use any other model.

What you expected to happen

inpainting should work

How to reproduce the problem

No response

Additional context

No response

Discord username

No response

grunblatt-git commented 3 months ago

This error seems to be dependent on the scheduler - i get the same error consistently with Euler, Euler a & Heun. I haven't noticed this to behave differently for different models yet.

psychedelicious commented 3 months ago

@grunblatt-git Are you on a mac too?

Assuming you are both on macOS, it's possible that this could be resolved using the PYTORCH_ENABLE_MPS_FALLBACK environment variable. This tells torch to use CPU when a specific operation isn't available on macOS (there are many that are not available).

The invoke.sh script sets this for you, so if you are using that, then we definitely have a bug - possibly introduced by diffusers.

If you run the app using the invokeai-web command, please try running it like this and see if that fixes the issue:

export PYTORCH_ENABLE_MPS_FALLBACK=1; invokeai-web

Paging @Vargol @gogurtenjoyer in case they have run into this on MPS.

grunblatt-git commented 3 months ago

@psychedelicious yes I am on Mac too, however so far I started invoke exclusively with the invoke.sh script anyways. So I guess the torch fallback was already set.

Could you reproduce it already or do you need anything else to troubleshoot this?

psychedelicious commented 3 months ago

Thanks. I haven't tried this out on macOS yet, but given your experience I have no doubt it will be a problem for me too. This kind of issue pops up fairly regularly for MPS unfortunately.

I'm not confident in my ability to resolve this. It could be an issue with diffusers. I see one open issue with the same error message, but it's related to text-to-video: https://github.com/huggingface/diffusers/issues/2785

I'll defer to @Vargol and/or @gogurtenjoyer for better understanding of the problem and if we need to raise this upstream or not.

Vargol commented 3 months ago

My install is a little none standard at the moment, a nightly PyTorch and a quick hack to enable bfloat16 and I've not noticed this issue. I've run a few inpainting tests with SDXL base and Euler that worked okay.

I've revert back to float16 and if it still works I'll knock up 'pure' 4.0rc environment to have a quick test.

Vargol commented 3 months ago

Okay...clean environment I get the error.

SDXL base with Euler scheduler - inpainting.

[2024-03-25 09:53:00,026]::[InvokeAI]::ERROR --> Error while invoking session 56557ff5-78f8-4c83-a00f-825b85bb59f1, invocation 0e4a70d0-44f5-4800-91db-ecddeab0909e (denoise_latents):
expected scalar type float but found c10::Half
[2024-03-25 09:53:00,026]::[InvokeAI]::ERROR --> Traceback (most recent call last):
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/invokeai/app/services/session_processor/session_processor_default.py", line 160, in _process
    outputs = self._invocation.invoke_internal(
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/invokeai/app/invocations/baseinvocation.py", line 281, in invoke_internal
    output = self.invoke(context)
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/invokeai/app/invocations/latent.py", line 787, in invoke
    result_latents = pipeline.latents_from_embeddings(
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 354, in latents_from_embeddings
    latents = self.generate_latents_from_embeddings(
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 432, in generate_latents_from_embeddings
    step_output = self.step(
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 580, in step
    step_output.pred_original_sample = guidance(
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 89, in __call__
    return self.apply_mask(latents, t)
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 107, in apply_mask
    masked_input = torch.where(mask_bool, latents, mask_latents)
RuntimeError: expected scalar type float but found c10::Half
Vargol commented 3 months ago

A bit more testing, it seems to be fixed in whatever version of 2.3.0 is in the test channel (2.2.2 in the same channel is a bust).

So

pip uninstall torch torchvision
pip3 install --pre torch==2.3.0 torchvision --index-url https://download.pytorch.org/whl/test/cpu

should work around the issue if nothing can be done in the apply_mask function.

pjparkjd commented 2 months ago

I am getting the same error after installing the latest version today. I don't know if these additional details will help, but I do launch using invoke.sh. The above fix did not work for me, but switching to the PNDM schedule did.

Description

Attempting to expand an image's aspect ratio using Unified Canvas results in a RuntimeError. This occurs with both the Cyberrealistic_v4.1_FP16 and Juggernaut-XL-v9 models when invoking the model after setting a mask.

Environment

Steps to Reproduce

  1. Select "Unified Canvas" mode.
  2. Upload an image.
  3. Adjust the canvas to a wider width.
  4. Set "Base/Mask" dropdown to "Mask".
  5. Use the Brush tool to paint over the areas to keep.
  6. Click "Invoke".

Expected Behavior

The image should be expanded to the specified aspect ratio without altering the masked area.

Actual Behavior

The process fails with a RuntimeError indicating a type mismatch: "expected scalar type float but found c10::Half".

Error Message

Error Traceback (most recent call last): File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/invokeai/app/services/session_processor/session_processor_default.py", line 185, in _process outputs = self._invocation.invoke_internal( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/invokeai/app/invocations/baseinvocation.py", line 281, in invoke_internal output = self.invoke(context) ^^^^^^^^^^^^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/invokeai/app/invocations/latent.py", line 940, in invoke result_latents = pipeline.latents_from_embeddings( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 339, in latents_from_embeddings latents = self.generate_latents_from_embeddings( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 419, in generate_latents_from_embeddings step_output = self.step( ^^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 551, in step step_output.pred_original_sample = guidance( ^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 88, in __call__ return self.apply_mask(latents, t) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 106, in apply_mask masked_input = torch.where(mask_bool, latents, mask_latents) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: expected scalar type float but found c10::Half

Kaputmaher commented 2 months ago

Same was for M2 Max in Pinokio environment. Was fixed by: cd /Users/admin/pinokio/api/invoke.git/app source env/bin/activate pip uninstall torch torchvision pip install --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/cpu

deseven commented 3 weeks ago

Same exact error for me. None of the solutions above worked, I guess nightly and test channels moved forward since then. Maybe some specific version would work, but I'm not sure how to determine that.

M1, macOS Sonoma

Vargol commented 3 weeks ago

still working here with torch==2.3.1

deseven commented 3 weeks ago

With 2.3.1 I'm just getting a different kind of error (while simply trying to outpaint a 256x256 image:

[2024-06-24 21:41:44,220]::[InvokeAI]::ERROR --> Error while invoking session a916852f-3483-4481-8935-b6ac92ab2b94, invocation 27260781-24a0-4fa9-9831-fc3551069040 (canvas_paste_back): images do not match
[2024-06-24 21:41:44,220]::[InvokeAI]::ERROR --> Traceback (most recent call last):
  File "/Users/deseven/invokeai/.venv/lib/python3.11/site-packages/invokeai/app/services/session_processor/session_processor_default.py", line 129, in run_node
    output = invocation.invoke_internal(context=context, services=self._services)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deseven/invokeai/.venv/lib/python3.11/site-packages/invokeai/app/invocations/baseinvocation.py", line 297, in invoke_internal
    return self.invoke(context)
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/deseven/invokeai/.venv/lib/python3.11/site-packages/invokeai/app/invocations/image.py", line 954, in invoke
    source_image.paste(target_image, (0, 0), mask)
  File "/Users/deseven/invokeai/.venv/lib/python3.11/site-packages/PIL/Image.py", line 1760, in paste
    self.im.paste(im, box, mask.im)
ValueError: images do not match
Vargol commented 3 weeks ago

Thats a different bug, you'll need to raise it as a new issue.

smile4yourself commented 3 weeks ago

I also can not get the canvas to work; same error "expected scalar type float but found c10::Half; running v4.2.4 on a Mac Ultra, using the install script to recreate the working folder but over-writing the previous version so that I still have my photos in the new version.

Vargol commented 3 weeks ago

Have you tried using torch 2.3.1 ? from the dev console or activated venv run

pip uninstall torch torchvision
pip install torch torchvision

that should install torch 2.3.1 assuming its supported on you macOS version

Ignore the warning about them being incompatible.

If it doesn't work then the devs will need all the details, what model, scheduler. steps, infill etc.

smile4yourself commented 2 weeks ago

Thanks so much! That fixed it even though I apparently have other issues. I include my console log fyi below

What would you like to do?
1: Generate images using the browser-based interface
2: Open the developer console
3: Command-line help
Q: Quit

To update, download and run the installer from https://github.com/invoke-ai/InvokeAI/releases/latest.

Please enter 1-4, Q: [1] 2
Open the developer console
Python 3.10.14
Press ^D to exit

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.

(InvokeAI) MAC_at_home@Mac-Studio ~/invoke4> pip uninstall torch torchvision
Found existing installation: torch 2.2.2
Uninstalling torch-2.2.2:
  Would remove:
    /Users/MAC_at_home/invoke4/.venv/bin/convert-caffe2-to-onnx
    /Users/MAC_at_home/invoke4/.venv/bin/convert-onnx-to-caffe2
    /Users/MAC_at_home/invoke4/.venv/bin/torchrun
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/functorch/*
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/torch-2.2.2.dist-info/*
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/torch/*
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/torchgen/*
  Would not remove (might be manually added):
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/torch/.DS_Store
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/torch/include/.DS_Store
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/torch/include/ATen/.DS_Store
Proceed (Y/n)? Y

  Successfully uninstalled torch-2.2.2

Found existing installation: torchvision 0.17.2
Uninstalling torchvision-0.17.2:
  Would remove:
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/torchvision-0.17.2.dist-info/*
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/torchvision/*
Proceed (Y/n)? y

  Successfully uninstalled torchvision-0.17.2

(InvokeAI) MAC_at_home@Mac-Studio ~/invoke4> pip install torch torchvision

Collecting torch
  Downloading torch-2.3.1-cp311-none-macosx_11_0_arm64.whl.metadata (26 kB)
Collecting torchvision
  Downloading torchvision-0.18.1-cp311-cp311-macosx_11_0_arm64.whl.metadata (6.6 kB)
Requirement already satisfied: filelock in ./.venv/lib/python3.11/site-packages (from torch) (3.15.1)
Requirement already satisfied: typing-extensions>=4.8.0 in ./.venv/lib/python3.11/site-packages (from torch) (4.12.2)
Requirement already satisfied: sympy in ./.venv/lib/python3.11/site-packages (from torch) (1.12.1)
Requirement already satisfied: networkx in ./.venv/lib/python3.11/site-packages (from torch) (3.3)
Requirement already satisfied: jinja2 in ./.venv/lib/python3.11/site-packages (from torch) (3.1.4)
Requirement already satisfied: fsspec in ./.venv/lib/python3.11/site-packages (from torch) (2024.3.1)
Requirement already satisfied: numpy in ./.venv/lib/python3.11/site-packages (from torchvision) (1.26.4)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in ./.venv/lib/python3.11/site-packages (from torchvision) (10.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./.venv/lib/python3.11/site-packages (from jinja2->torch) (2.1.5)
Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in ./.venv/lib/python3.11/site-packages (from sympy->torch) (1.3.0)
Downloading torch-2.3.1-cp311-none-macosx_11_0_arm64.whl (61.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.0/61.0 MB 61.9 MB/s eta 0:00:00
Downloading torchvision-0.18.1-cp311-cp311-macosx_11_0_arm64.whl (1.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 49.9 MB/s eta 0:00:00
Installing collected packages: torch, torchvision

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
invokeai 4.2.4 requires torch==2.2.2, but you have torch 2.3.1 which is incompatible.
invokeai 4.2.4 requires torchvision==0.17.2, but you have torchvision 0.18.1 which is incompatible.

Successfully installed torch-2.3.1 torchvision-0.18.1

[notice] A new release of pip is available: 24.1 -> 24.1.1
[notice] To update, run: pip install --upgrade pip

(InvokeAI) MAC_at_home@Mac-Studio ~/invoke4>
hsm207 commented 1 week ago

I had the same issue too after upgrading my developer install to commit 35f8781ea22cccbbbb6fab92180ee2b7bca35a70.

Upgrading to pytorch 2.3.1 from 2.2.2 fixed it for me.

smile4yourself commented 3 days ago

After upgrading invokeAI to 4.2.6post1 on the MAC, this error came back. I resolved it the same way as you suggested before, 

pip uninstall torch torchvision
pip install torch torchvision

regards