Image2Image, Image Prompt, and Inpainting all generate an error from comfyui package

[astralarkitekt]

Hi illya,

Foooocus is changing my life! :)

I'm having issues with a failure when attempting to use Upscale/Variant or Inpaint/Outpaint. It appears to be a problem with a member of the comfyui repo sd1_clip.py not getting a 'cpu' or 'cuda' value as expected on line 150 of that file. Here's the traceback:

[Parameters] Adaptive CFG = 7
[Parameters] Sharpness = 2
[Parameters] ADM Scale = 1.5 : 0.8 : 0.3
[Parameters] CFG = 7.0
[Fooocus] Downloading inpainter ...
[Parameters] Sampler = dpmpp_fooocus_2m_sde_inpaint_seamless - karras
[Parameters] Steps = 30 - 20
[Fooocus] Initializing ...
[Fooocus] Loading models ...
LoRAs loaded: [('sd_xl_offset_example-lora_1.0.safetensors', 0.5), ('None', 0.5), ('None', 0.5), ('None', 0.5), ('None', 0.5), ('C:\\dev\\focus\\Fooocus\\models\\inpaint\\inpaint.fooocus.patch', 1.0)]
loading new
unload clone 0
loading in lowvram mode 256.0
loading in lowvram mode 256.0
[Fooocus] Processing prompts ...
[Fooocus] Preparing Fooocus text #1 ...
Fooocus Expansion loaded by itself.
[Prompt Expansion] New suffix: extremely high detail, photo realistic, cinematic lighting, pen and ink, intricate line drawings, by Yoshitaka Amano, Ruan Jia, Kentaro Miura, Artgerm, post processed, concept art, artstation, matte painting, style by eddie mendoza, raphael lacoste, alex ross
[Fooocus] Preparing Fooocus text #2 ...
Fooocus Expansion loaded by itself.
[Prompt Expansion] New suffix: intricate, horror, highly detailed, digital painting, artstation, concept art, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha
[Fooocus] Encoding positive #1 ...
loading new
loading in lowvram mode 256.0
Traceback (most recent call last):
  File "C:\dev\focus\Fooocus\modules\async_worker.py", line 565, in worker
    handler(task)
  File "C:\dev\focus\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\dev\focus\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\dev\focus\Fooocus\modules\async_worker.py", line 269, in handler
    t['c'] = pipeline.clip_encode(texts=t['positive'], pool_top_k=positive_top_k)
  File "C:\dev\focus\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\dev\focus\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\dev\focus\Fooocus\modules\default_pipeline.py", line 177, in clip_encode
    cond, pooled = clip_encode_single(final_clip, text)
  File "C:\dev\focus\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\dev\focus\python_embeded\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\dev\focus\Fooocus\modules\default_pipeline.py", line 154, in clip_encode_single
    result = clip.encode_from_tokens(tokens, return_pooled=True)
  File "C:\dev\focus\Fooocus\repositories\ComfyUI-from-StabilityAI-Official\comfy\sd.py", line 120, in encode_from_tokens
    cond, pooled = self.cond_stage_model.encode_token_weights(tokens)
  File "C:\dev\focus\Fooocus\repositories\ComfyUI-from-StabilityAI-Official\comfy\sdxl_clip.py", line 56, in encode_token_weights
    g_out, g_pooled = self.clip_g.encode_token_weights(token_weight_pairs_g)
  File "C:\dev\focus\Fooocus\modules\patch.py", line 246, in encode_token_weights_patched_with_a1111_method
    out, pooled = self.encode(to_encode)
  File "C:\dev\focus\Fooocus\repositories\ComfyUI-from-StabilityAI-Official\comfy\sd1_clip.py", line 179, in encode
    return self(tokens)
  File "C:\dev\focus\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\dev\focus\python_embeded\lib\site-packages\accelerate\hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "C:\dev\focus\Fooocus\repositories\ComfyUI-from-StabilityAI-Official\comfy\sd1_clip.py", line 150, in forward
    with precision_scope(model_management.get_autocast_device(device), torch.float32):
  File "C:\dev\focus\python_embeded\lib\site-packages\torch\amp\autocast_mode.py", line 201, in __init__
    raise RuntimeError('User specified autocast device_type must be \'cuda\' or \'cpu\'')
RuntimeError: User specified autocast device_type must be 'cuda' or 'cpu'

Here's my system stats from Foooocus boot output:

Already up-to-date
Update succeeded.
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Fooocus version: 2.1.35
Inference Engine exists and URL is correct.
Inference Engine checkout finished for d1a0abd40b86f3f079b0cc71e49f9f4604831457.
Running on local URL:  http://127.0.0.1:8188

To create a public link, set `share=True` in `launch()`.
Total VRAM 4096 MB, total RAM 32389 MB
Trying to enable lowvram mode because your GPU seems to have 4GB or less. If you don't want this use: --normalvram
xformers version: 0.0.20
Set vram state to: LOW_VRAM
Device: cuda:0 NVIDIA GeForce RTX 3050 Laptop GPU : native
VAE dtype: torch.bfloat16
Using xformers cross attention
model_type EPS
adm 2560
Refiner model loaded: C:\dev\focus\Fooocus\models\checkpoints\sd_xl_refiner_1.0_0.9vae.safetensors
model_type EPS
adm 2816
making attention of type 'vanilla-xformers' with 512 in_channels
building MemoryEfficientAttnBlock with 512 in_channels...
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla-xformers' with 512 in_channels
building MemoryEfficientAttnBlock with 512 in_channels...
missing {'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids', 'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_l.text_projection'}
Base model loaded: C:\dev\focus\Fooocus\models\checkpoints\sd_xl_base_1.0_0.9vae.safetensors
LoRAs loaded: [('sd_xl_offset_example-lora_1.0.safetensors', 0.5), ('None', 0.5), ('None', 0.5), ('None', 0.5), ('None', 0.5)]
Fooocus Expansion engine loaded for cuda:0, use_fp16 = True.
loading new
loading in lowvram mode 256.0
loading in lowvram mode 256.0
App started successful. Use the app with http://127.0.0.1:8188/ or 127.0.0.1:8188

[astralarkitekt]

Update: It's happening with all prompt types, text, image both.

[lllyasviel]

see if fixed in 2.1.37

[astralarkitekt]

testing now, thank youuuuuu!

On Tue, Oct 10, 2023 at 4:36 AM lllyasviel @.***> wrote:

see if fixed in 2.1.37

— Reply to this email directly, view it on GitHub https://github.com/lllyasviel/Fooocus/issues/620#issuecomment-1754824803, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADLKN3TILIRB22P72VRSO3X6UJLPAVCNFSM6AAAAAA52AI3QWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJUHAZDIOBQGM . You are receiving this because you authored the thread.Message ID: @.***>

[astralarkitekt]

Verified that it is no longer spitting out that error in any of the modes I was experiencing it in before. I believe you can close this now, but will ping this ticket if the problem recurs.

Check it out!