C4l1b3r
commented
4 months ago When you load it up in the beginning does it says deforum controlnet activated? Mine doesn't would love a fix as well
Closed Vendaciousness closed 3 months ago
C4l1b3r
commented
4 months ago When you load it up in the beginning does it says deforum controlnet activated? Mine doesn't would love a fix as well
Vendaciousness
commented
4 months ago When you load it up in the beginning does it says deforum controlnet activated? Mine doesn't would love a fix as well
I did not see that, no. Speaking of that, I didn't include an example output from the CMD window:
venv "J:\Code-2\stable-diffusion-webui-forge\venv\Scripts\Python.exe"
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec 6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Version: f0.0.17v1.8.0rc-latest-276-g29be1da7
Commit hash: 29be1da7cf2b5dccfc70fbdd33eb35c56a31ffb7
Launching Web UI with arguments: --ckpt-dir J:\SD_Models\SDXL_Models\checkpoints --embeddings-dir J:\SD_Models\SDXL_Models\embeddings --lora-dir J:\SD_Models\SDXL_Models\lora
Total VRAM 24576 MB, total RAM 65462 MB
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 3090 : native
Hint: your device supports --pin-shared-memory for potential speed improvements.
Hint: your device supports --cuda-malloc for potential speed improvements.
Hint: your device supports --cuda-stream for potential speed improvements.
VAE dtype: torch.bfloat16
CUDA Stream Activated: False
Using pytorch cross attention
ControlNet preprocessor location: J:\Code-2\stable-diffusion-webui-forge\models\ControlNetPreprocessor
Loading weights [4496b36d48] from J:\SD_Models\SDXL_Models\checkpoints\dreamshaperXL_v21TurboDPMSDE.safetensors
2024-03-30 22:51:33,377 - ControlNet - INFO - ControlNet UI callback registered.
model_type EPS
UNet ADM Dimension 2816
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
Startup time: 19.4s (prepare environment: 4.0s, import torch: 6.0s, import gradio: 1.4s, setup paths: 1.6s, initialize shared: 0.2s, other imports: 0.7s, load scripts: 3.8s, create ui: 1.0s, gradio launch: 0.6s).
Using pytorch attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using pytorch attention in VAE
extra {'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids', 'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids', 'cond_stage_model.clip_l.text_projection'}
left over keys: dict_keys(['conditioner.embedders.0.logit_scale', 'conditioner.embedders.0.text_projection'])
Loading VAE weights specified in settings: J:\Code-2\stable-diffusion-webui-forge\models\VAE\fixFP16ErrorsSDXLLowerMemoryUse_v10.safetensors
To load target model SDXLClipModel
Begin to load 1 model
[Memory Management] Current Free GPU Memory (MB) = 23318.37109375
[Memory Management] Model Memory (MB) = 2144.3546981811523
[Memory Management] Minimal Inference Memory (MB) = 1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) = 20150.016395568848
Moving model(s) has taken 0.48 seconds
Model loaded in 7.7s (load weights from disk: 0.7s, forge load real models: 5.3s, load VAE: 0.6s, calculate empty prompt: 0.9s).
reading custom settings from J:\Code-2\stable-diffusion-webui-forge\deforum_settings.txt-V
rewbs
commented
3 months ago I have pushed a fix for #6 which is most likely the same root cause as what you're seeing here. Could you please update your extension and take it for a spin & let me know how it goes? Thanks!
Vendaciousness
commented
3 months ago Here's what I got. I ran a few kinds of models, just to see if they behaved differently:
started run_deforum
times_to_run: 1
Deforum extension for Forge webui
Git commit: Unknown
Starting job task(ey83twu9v7xb6i5)-0...
Unpacking ControlNet 1 base video
Exporting Video Frames to J:\AI-OUT-A11\Deforum-Live_Anywahere-PNW-Cabin-1\controlnet_1_inputframes...
Trying to extract frames from video with input FPS of 15.031315240083508. Please wait patiently.
Extracted 480 frames from video in 1.94 seconds!
Loading 480 input frames from J:\AI-OUT-A11\Deforum-Live_Anywahere-PNW-Cabin-1\controlnet_1_inputframes and saving video frames to J:\AI-OUT-A11\Deforum-Live_Anywahere-PNW-Cabin-1
ControlNet 1 base video unpacked!
Saving animation frames to:
J:\AI-OUT-A11\Deforum-Live_Anywahere-PNW-Cabin-1
Animation frame: 0/480
Seed: 949975291
Prompt: (a little square lookout-style fortified tiny home tower mounted on plinths in a barren wasteland) (tiny house:2), (small, 1 bed 1 bath:2), (made entirely from concrete and metal :1.3), (watercolor and ink outliner on paper), a small path leads to a single-room cube, dwarfed by the surrounding scenery, (Neo-futurism:1.3), (securely attached to the side of sheer rock face of a granite canyon hundreds of feet above canadian forest), (surrounded by steep cliffs and a scenic mountains), showcasing its construction techniques and design elements, (remote location), ((masterpiece)), ((best quality)), ((high detail)), (colored pencil and watercolor brush texture:1.5), (architectural concept drawing:1.5), (slab-like solid construction and heavy-duty support beams:1.5) <lora:stylesqueeze_v3_xl:0.5> <lora:RMSDXL_Enhance:0.8> <lora:Style_Watercolor_and_Ink_v1_SDXL_LoRA:0.6> <lora:HandSketcher-XL:0.6>
Neg Prompt: ((stilts)), (wood, wooden deck, stilts, mansion, Deck Joists, Horizontal beams that support the decking boards, scaffolding:2), (googie), (space age 60's architecture) (scaffolding:1.7), sandstone, red rock, half-dome, multiple houses, asymmetric, uneven, oblong, ugly, organic, signature, text, writing, watermark, multiple houses, asymmetric, uneven, oblong, ugly, figures, animals, distortions, noise, artifacts, low resolution, watermark, logos, text, poor lighting, out of focus, unnatural colors, flat, overexposed, underexposed, pixelated, composition, framing, contrast, color grading, atmospheric effects, storytelling, immersive people, person, man, woman, child, boy, girl, humanoid, character (googie), (space age 60's architecture) (scaffolding:1.7), sandstone, red rockhalf-dome, multiple houses, asymmetric, uneven, oblong, ugly, organic,
Not using an init image (doing pure txt2img)
╭─────┬───┬───────┬────┬────┬────┬────┬────┬────╮
│Steps│CFG│Denoise│Tr X│Tr Y│Tr Z│Ro X│Ro Y│Ro Z│
├─────┼───┼───────┼────┼────┼────┼────┼────┼────┤
│ 30 │3.0│ 0 │ 2 │ 0 │ 0 │ 0 │ 0 │ 0 │
╰─────┴───┴───────┴────┴────┴────┴────┴────┴────╯
Reading ControlNet 1 base frame #0 at J:\AI-OUT-A11\Deforum-Live_Anywahere-PNW-Cabin-1\controlnet_1_inputframes\000000000.jpg
ControlNet 1: weight=0.0, guidance_start=0.0, guidance_end=1.0
Added script: ControlNet with 5 args, at positions 0-5 (of 0-4.)
2024-04-09 13:10:51,615 - ControlNet - INFO - ControlNet Input Mode: InputMode.SIMPLE
*** Error running process: J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "J:\Code-2\stable-diffusion-webui-forge\modules\scripts.py", line 803, in process
script.process(p, *script_args)
File "J:\Code-2\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 548, in process
self.process_unit_after_click_generate(p, unit, params, *args, **kwargs) File "J:\Code-2\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 298, in process_unit_after_click_generate input_list, resize_mode = self.get_input_data(p, unit, preprocessor, h, w)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 227, in get_input_data
image = self.try_crop_image_with_a1111_mask(p, unit, image, resize_mode, preprocessor)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 126, in try_crop_image_with_a1111_mask
preprocessor.corp_image_with_a1111_mask_when_in_img2img_inpaint_tab
AttributeError: 'NoneType' object has no attribute 'corp_image_with_a1111_mask_when_in_img2img_inpaint_tab'
---
*** Error running process_before_every_sampling: J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "J:\Code-2\stable-diffusion-webui-forge\modules\scripts.py", line 835, in process_before_every_sampling
script.process_before_every_sampling(p, *script_args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 555, in process_before_every_sampling
self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs)
KeyError: 0
---
100%|██████████████████████████████████████████| 30/30 [00:27<00:00, 1.11it/s]
*** Error running postprocess_batch_list: J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "J:\Code-2\stable-diffusion-webui-forge\modules\scripts.py", line 859, in postprocess_batch_list
script.postprocess_batch_list(p, pp, *script_args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 561, in postprocess_batch_list
self.process_unit_after_every_sampling(p, unit, self.current_params[i], pp, *args, **kwargs)
KeyError: 0
---
render preview video: frame_idx=1 preview_interval_frames=100 anim_args.max_frames=480 is_preview_on=False is_preview_frame=False is_close_to_end=False
No preview video on frame 1.
Animation frame: 1/480
Seed: 949975292
Prompt: (a little square lookout-style fortified tiny home tower mounted on plinths in a barren wasteland) (tiny house:2), (small, 1 bed 1 bath:2), (made entirely from concrete and metal :1.3), (watercolor and ink outliner on paper), a small path leads to a single-room cube, dwarfed by the surrounding scenery, (Neo-futurism:1.3), (securely attached to the side of sheer rock face of a granite canyon hundreds of feet above canadian forest), (surrounded by steep cliffs and a scenic mountains), showcasing its construction techniques and design elements, (remote location), ((masterpiece)), ((best quality)), ((high detail)), (colored pencil and watercolor brush texture:1.5), (architectural concept drawing:1.5), (slab-like solid construction and heavy-duty support beams:1.5) <lora:stylesqueeze_v3_xl:0.5> <lora:RMSDXL_Enhance:0.8> <lora:Style_Watercolor_and_Ink_v1_SDXL_LoRA:0.6> <lora:HandSketcher-XL:0.6>
Neg Prompt: ((stilts)), (wood, wooden deck, stilts, mansion, Deck Joists, Horizontal beams that support the decking boards, scaffolding:2), (googie), (space age 60's architecture) (scaffolding:1.7), sandstone, red rock, half-dome, multiple houses, asymmetric, uneven, oblong, ugly, organic, signature, text, writing, watermark, multiple houses, asymmetric, uneven, oblong, ugly, figures, animals, distortions, noise, artifacts, low resolution, watermark, logos, text, poor lighting, out of focus, unnatural colors, flat, overexposed, underexposed, pixelated, composition, framing, contrast, color grading, atmospheric effects, storytelling, immersive people, person, man, woman, child, boy, girl, humanoid, character (googie), (space age 60's architecture) (scaffolding:1.7), sandstone, red rockhalf-dome, multiple houses, asymmetric, uneven, oblong, ugly, organic,
╭─────┬───┬───────┬──────┬────┬────┬────┬────┬────╮
│Steps│CFG│Denoise│ Tr X │Tr Y│Tr Z│Ro X│Ro Y│Ro Z│
├─────┼───┼───────┼──────┼────┼────┼────┼────┼────┤
│ 30 │3.0│0.7951 │1.9728│ 0 │ 0 │ 0 │ 0 │ 0 │
╰─────┴───┴───────┴──────┴────┴────┴────┴────┴────╯
Reading ControlNet 1 base frame #1 at J:\AI-OUT-A11\Deforum-Live_Anywahere-PNW-Cabin-1\controlnet_1_inputframes\000000001.jpg
ControlNet 1: weight=0.12004467412919506, guidance_start=0.0, guidance_end=1.0
Added script: ControlNet with 5 args, at positions 0-5 (of 0-4.)
2024-04-09 13:11:21,179 - ControlNet - INFO - ControlNet Input Mode: InputMode.SIMPLE
*** Error running process: J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "J:\Code-2\stable-diffusion-webui-forge\modules\scripts.py", line 803, in process
script.process(p, *script_args)
File "J:\Code-2\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 548, in process
self.process_unit_after_click_generate(p, unit, params, *args, **kwargs) File "J:\Code-2\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 298, in process_unit_after_click_generate input_list, resize_mode = self.get_input_data(p, unit, preprocessor, h, w)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 227, in get_input_data
image = self.try_crop_image_with_a1111_mask(p, unit, image, resize_mode, preprocessor)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 126, in try_crop_image_with_a1111_mask
preprocessor.corp_image_with_a1111_mask_when_in_img2img_inpaint_tab
AttributeError: 'NoneType' object has no attribute 'corp_image_with_a1111_mask_when_in_img2img_inpaint_tab'
---
*** Error running process_before_every_sampling: J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "J:\Code-2\stable-diffusion-webui-forge\modules\scripts.py", line 835, in process_before_every_sampling
script.process_before_every_sampling(p, *script_args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 555, in process_before_every_sampling
self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs)
KeyError: 0
---
42%|█████████████████▌ | 10/24 [00:06<00:08, 1.61it/s]
*** Error running postprocess_batch_list: J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "J:\Code-2\stable-diffusion-webui-forge\modules\scripts.py", line 859, in postprocess_batch_list
script.postprocess_batch_list(p, pp, *script_args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 561, in postprocess_batch_list
self.process_unit_after_every_sampling(p, unit, self.current_params[i], pp, *args, **kwargs)
KeyError: 0
---
render preview video: frame_idx=2 preview_interval_frames=100 anim_args.max_frames=480 is_preview_on=False is_preview_frame=False is_close_to_end=False
No preview video on frame 2.
Animation frame: 2/480
Seed: 949975293
Prompt: (a little square lookout-style fortified tiny home tower mounted on plinths in a barren wasteland) (tiny house:2), (small, 1 bed 1 bath:2), (made entirely from concrete and metal :1.3), (watercolor and ink outliner on paper), a small path leads to a single-room cube, dwarfed by the surrounding scenery, (Neo-futurism:1.3), (securely attached to the side of sheer rock face of a granite canyon hundreds of feet above canadian forest), (surrounded by steep cliffs and a scenic mountains), showcasing its construction techniques and design elements, (remote location), ((masterpiece)), ((best quality)), ((high detail)), (colored pencil and watercolor brush texture:1.5), (architectural concept drawing:1.5), (slab-like solid construction and heavy-duty support beams:1.5) <lora:stylesqueeze_v3_xl:0.5> <lora:RMSDXL_Enhance:0.8> <lora:Style_Watercolor_and_Ink_v1_SDXL_LoRA:0.6> <lora:HandSketcher-XL:0.6>
Neg Prompt: ((stilts)), (wood, wooden deck, stilts, mansion, Deck Joists, Horizontal beams that support the decking boards, scaffolding:2), (googie), (space age 60's architecture) (scaffolding:1.7), sandstone, red rock, half-dome, multiple houses, asymmetric, uneven, oblong, ugly, organic, signature, text, writing, watermark, multiple houses, asymmetric, uneven, oblong, ugly, figures, animals, distortions, noise, artifacts, low resolution, watermark, logos, text, poor lighting, out of focus, unnatural colors, flat, overexposed, underexposed, pixelated, composition, framing, contrast, color grading, atmospheric effects, storytelling, immersive people, person, man, woman, child, boy, girl, humanoid, character (googie), (space age 60's architecture) (scaffolding:1.7), sandstone, red rockhalf-dome, multiple houses, asymmetric, uneven, oblong, ugly, organic,
Deforum progress: 1%|▏ | 40/4669 [00:29<56:28, 1.37it/s]
Got a request to stitch frames to video using FFmpeg.
Frames:
J:\AI-OUT-A11\Deforum-Live_Anywahere-PNW-Cabin-1\20240409131048_%09d.png
To Video:
J:\AI-OUT-A11\Deforum-Live_Anywahere-PNW-Cabin-1\20240409131048.mp4
Stitching *video*...
Video stitching done in 1.18 seconds!Looks like Illyasviel may have made some changes under the hood to how CN interacts with A1111, if lines like these are any indication:
line 126, in try_crop_image_with_a1111_mask
preprocessor.corp_image_with_a1111_mask_when_in_img2img_inpaint_tab
AttributeError: 'NoneType' object has no attribute 'corp_image_with_a1111_mask_when_in_img2img_inpaint_tab'Sorry I could give you better news, Rewbs. Let me know if there's anything else I can do.
-v
rewbs
commented
3 months ago Could you share your settings file so I can see the exact controlnet config? Thanks!
C4l1b3r
commented
3 months ago Update worked for me
Vendaciousness
commented
3 months ago That very useful, thanks @C4l1b3r
Here's my settings file. I tried using two different SDXL ControlNet models for the errors in the cap: An Inpaint model: https://huggingface.co/destitech/controlnet-inpaint-dreamer-sdxl/tree/main And QR Code monster: https://huggingface.co/monster-labs/control_v1p_sdxl_qrcode_monster Here's my settings file: 20240409132813_settings.txt
I'm using some special settings in my webui-user.bat, in case that's important. Here's webui-user.bat: (it won't let me uploat a bat file..)
@echo off
set SAFETENSORS_FAST_GPU=1
set CUDA_VISIBLE_DEVICES=0
set PYTHON=
set PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.9,max_split_size_mb:512
set A1111_HOME= "J:\SD_Models\SDXL_Models"
set COMMANDLINE_ARGS=%COMMANDLINE_ARGS% ^
--ckpt-dir "J:\SD_Models\SDXL_Models\checkpoints" ^
--embeddings-dir "J:\SD_Models\SDXL_Models\embeddings" ^
--lora-dir "J:\SD_Models\SDXL_Models\lora"
call webui.batSorry if it's some stupid setting I have set... -v
Vendaciousness
commented
3 months ago Ok, I got some free time to do some more testing and figured out what breaks it: using 'None' for the Preprocessor.
Rewbs, what you said about only incorrect preprocessors causing errors for you helped me clue in to it. Thank you! Is 'None' no longer supported in SF? I was just looking in their Issues and found the 'TypeError: 'NoneType' object is not iterable' error happens over there, too, so this might not be a Deforum thing at all.
If you're wondering why I'd ever use 'None', here's some background: I actually use 'None' more than anything else, since I often create my own depth map videos for inputs, since I can't get the output I need from the preprocessors.
Same goes for OpenPose, since the preprocessors, if you look at the output, get body position wrong at least a few frames every time, so the only way I've gotten consistent tracking is by either creating an OpenPose render in Blender, or by enabling 'Allow Preview' check box in the CN txt2img panel to get the 'Preprocessor Preview', which I then fix with the OpenPose Editor, save and then assemble into a video.
Ideally, I use 'None' for every video I make, along with an Inpaint model, which works in vanilla A1111. Then again, Forge has support in txt2img and img2img for the much improved FoOocus Inpainting models, which are here: https://huggingface.co/lllyasviel/fooocus_inpaint/tree/main
These would be super cool to use in Deforum and may even allow for good masking from within Deforum.
Vendaciousness
commented
3 months ago @rewbs I have good news on this bug! Turns out ControlNet SDXL Inpaint is functional in txt2img and img2img, just not in every situation. I changed some batch flags and now it works, but I'm not able to pinpoint what fixes it yet.
So far I know this much:
If you let img2img send an image to ControlNet, rather than check 'Upload independent control image', you get the same kind of error I get in Deforum when I try the same models.
In other words, if you use the what their doing when the option is checked, it should fix our issue.
It works even with the preprocessor set to 'None'. The Inpaint-Dreamer CN model also runs without errors, but the Inpainted area comes out black. For anyone getting the earlier errors, the only things I changed since last week was adding these flags to me webui-user.bat:
--pin-shared-memory ^
--cuda-malloc ^
--cuda-stream ^Here's the error I get in the img2img tab when I try to use the FoOocus Inpaint model linked below, with any Inpaint preprocessor, or set to 'None':
To load target model SDXLClipModel
Begin to load 1 model
Reuse 1 loaded models
[Memory Management] Current Free GPU Memory (MB) = 17738.81005859375
[Memory Management] Model Memory (MB) = 1362.0
[Memory Management] Minimal Inference Memory (MB) = 1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) = 15352.81005859375
Moving model(s) has taken 0.81 seconds
*** Error running process_before_every_sampling: J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "J:\Code-2\stable-diffusion-webui-forge\modules\scripts.py", line 835, in process_before_every_sampling
script.process_before_every_sampling(p, *script_args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 555, in process_before_every_sampling
self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 501, in process_unit_before_every_sampling
params.model.process_before_every_sampling(p, cond, mask, *args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_fooocus_inpaint\scripts\forge_fooocus_inpaint.py", line 80, in process_before_every_sampling
latent_mask = torch.nn.functional.max_pool2d(mask_original, (8, 8)).round().to(cond)
File "J:\Code-2\stable-diffusion-webui-forge\venv\lib\site-packages\torch\_jit_internal.py", line 488, in fn
return if_false(*args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\venv\lib\site-packages\torch\nn\functional.py", line 791, in _max_pool2d
return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
TypeError: max_pool2d(): argument 'input' (position 1) must be Tensor, not NoneTypeNow this one here is the error I get when I try to run the FoOocus Inpaint model from ControlNet in Deforum:
To load target model SDXLClipModel
Begin to load 1 model
Reuse 1 loaded models
[Memory Management] Current Free GPU Memory (MB) = 15279.814022064209
[Memory Management] Model Memory (MB) = 0.0
[Memory Management] Minimal Inference Memory (MB) = 1024.0
[Memory Management] Estimated Remaining GPU Memory (MB) = 14255.814022064209
Moving model(s) has taken 0.83 seconds
*** Error running process_before_every_sampling: J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "J:\Code-2\stable-diffusion-webui-forge\modules\scripts.py", line 835, in process_before_every_sampling
script.process_before_every_sampling(p, *script_args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 555, in process_before_every_sampling
self.process_unit_before_every_sampling(p, unit, self.current_params[i], *args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_controlnet\scripts\controlnet.py", line 501, in process_unit_before_every_sampling
params.model.process_before_every_sampling(p, cond, mask, *args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\extensions-builtin\sd_forge_fooocus_inpaint\scripts\forge_fooocus_inpaint.py", line 80, in process_before_every_sampling
latent_mask = torch.nn.functional.max_pool2d(mask_original, (8, 8)).round().to(cond)
File "J:\Code-2\stable-diffusion-webui-forge\venv\lib\site-packages\torch\_jit_internal.py", line 488, in fn
return if_false(*args, **kwargs)
File "J:\Code-2\stable-diffusion-webui-forge\venv\lib\site-packages\torch\nn\functional.py", line 791, in _max_pool2d
return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
TypeError: max_pool2d(): argument 'input' (position 1) must be Tensor, not NoneTypeLook familiar? Hehe... I let you puzzle out whatever that means.
Here's the link again for the FoOocus Inpaint model which works in txt2img and img2im (with 'Upload independent control image' enabled)
-v
Vendaciousness
commented
3 months ago I figured out what causes the errors in img2img: Not drawing a mask or using a separate mask file.
So you could fix things in this repo just by adding the support for the 'Use Mask' functionality from regular CN.
I tried using a mask input video in the field labeled ;ControlNet Mask Video/ Image Path (NOT WORKING, kept in UI for CN's devs testing!)'. I just used a solid white video, but I got the same error as above and similar to what I get in img2img with no mask or mask image.
GevatterWuff
commented
3 months ago Just ran into the "None" Bug with an ip-adapter ControlNet. Since I didn't read the solution in this thread, here's my workaround: If you get "KeyError: 0" just click "None" again from the drop-down menu ... it works, so a fix should be a minor issue too.
Vendaciousness
commented
3 months ago Thanks, man! I've give it a shot!
What happened and why was it ungood?
I'm getting the error below when I try to use any SDXL ControlNet models within Deforum.
I tried the same CN SDXL models and settings in txt2img and img2img they both worked okay, so I thought I'd post my error, even though it looks quite similar to Issue #6, since that user I think was confused by some note in the code and stopped replying to rewbs.