search

brkirch / stable-diffusion-webui

Stable Diffusion web UI
GNU Affero General Public License v3.0
195 stars 10 forks source link

[Bug]: Experimental with Deforum 3D Mode with default options raises "RuntimeError: "compute_indices_weights_cubic" not implemented for 'Half'" #11

Open yyahav opened 1 year ago

yyahav commented 1 year ago

Is there an existing issue for this?

What happened?

Running Deforum 3D Mode with the experimental build using default command line options caused this error when trying to generate the second frame: RuntimeError: "compute_indices_weights_cubic" not implemented for 'Half'

Steps to reproduce the problem

  1. Install latest experimental build
  2. Install latest Deforum extension
  3. Run webui using the default options
  4. In Deforum extension Keyframes tab select 3D and Generate

What should have happened?

The video should have been generated properly without an error.

Commit where the problem happens

106c9ac5

What platforms do you use to access the UI ?

MacOS

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

--skip-torch-cuda-test --skip-install --no-download-sd-model --precision full --no-half-vae --upcast-sampling --opt-sub-quad-attention --use-cpu interrogate

List of extensions

deforum-for-automatic1111-webui sd-webui-controlnet

Console logs

/Users/username/Documents/GitHub/brkirch/stable-diffusion-webui/venv/bin/python3.10 /Users/username/Documents/GitHub/brkirch/stable-diffusion-webui/launch.py --skip-torch-cuda-test --skip-install --no-download-sd-model --precision full --no-half-vae --upcast-sampling --opt-sub-quad-attention --use-cpu interrogate 
Python 3.10.11 (main, Apr  7 2023, 07:24:53) [Clang 14.0.0 (clang-1400.0.29.202)]
Commit hash: 106c9ac512b94fab83c02514f9a0821673ac9442

Launching Web UI with arguments: --no-download-sd-model --precision full --no-half-vae --upcast-sampling --opt-sub-quad-attention --use-cpu interrogate
Warning: caught exception 'Torch not compiled with CUDA enabled', memory monitor disabled
No module 'xformers'. Proceeding without it.
/Users/username/Documents/GitHub/brkirch/stable-diffusion-webui/venv/lib/python3.10/site-packages/torchvision/transforms/functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in 0.15 and will be **removed in 0.17**. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional.
  warnings.warn(
Loading weights [1a189f0be6] from /Users/username/Documents/GitHub/brkirch/stable-diffusion-webui/models/Stable-diffusion/v1-5-pruned.safetensors
Creating model from config: /Users/username/Documents/GitHub/brkirch/stable-diffusion-webui/configs/v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Applying sub-quadratic cross attention optimization.
Textual inversion embeddings loaded(0): 
Model loaded in 3.3s (load weights from disk: 0.1s, create model: 0.6s, apply weights to model: 1.7s, apply half(): 0.3s, move model to device: 0.5s).
*Deforum ControlNet support: enabled*
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 8.4s (import torch: 1.1s, import gradio: 1.8s, import ldm: 0.3s, other imports: 0.7s, list extensions: 0.1s, load scripts: 0.8s, load SD checkpoint: 3.3s, create ui: 0.2s).
Deforum extension for auto1111 webui, v2.3b
Git commit: 4c0fdccc (Tue Apr 18 12:53:24 2023)
Saving animation frames to:
/Users/username/Documents/GitHub/brkirch/stable-diffusion-webui/outputs/img2img-images/Deforum_20230420182003
Loading MiDaS model...
Loading AdaBins model...
Using cache found in /Users/username/.cache/torch/hub/rwightman_gen-efficientnet-pytorch_master
Animation frame: 0/120  
Seed: 1760068600
Prompt: tiny cute swamp bunny, highly detailed, intricate, ultra hd, sharp photo, crepuscular rays, in focus, by tomasz alen kopera
Not using an init image (doing pure txt2img)
╭─────┬───┬───────┬────┬────┬────┬────┬────┬────╮
│Steps│CFG│Denoise│Tr X│Tr Y│Tr Z│Ro X│Ro Y│Ro Z│
├─────┼───┼───────┼────┼────┼────┼────┼────┼────┤
│ 25  │7.0│ None  │ 0  │ 0  │1.75│ 0  │ 0  │ 0  │
╰─────┴───┴───────┴────┴────┴────┴────┴────┴────╯
  4%|▍         | 1/25 [00:02<00:54,  2.28s/it]
Deforum progress:   0%|          | 0/556 [00:00<?, ?it/s]
 12%|█▏        | 3/25 [00:03<00:25,  1.16s/it]
Deforum progress:   1%|          | 3/556 [00:01<05:26,  1.69it/s]
 20%|██        | 5/25 [00:05<00:18,  1.07it/s]
Deforum progress:   1%|          | 5/556 [00:03<06:34,  1.40it/s]
 28%|██▊       | 7/25 [00:07<00:15,  1.17it/s]
Deforum progress:   1%|▏         | 7/556 [00:04<06:54,  1.32it/s]
 36%|███▌      | 9/25 [00:08<00:13,  1.22it/s]
Deforum progress:   2%|▏         | 9/556 [00:06<07:03,  1.29it/s]
 44%|████▍     | 11/25 [00:11<00:14,  1.04s/it]
Deforum progress:   2%|▏         | 11/556 [00:09<09:12,  1.01s/it]
 52%|█████▏    | 13/25 [00:12<00:10,  1.20it/s]
Deforum progress:   2%|▏         | 13/556 [00:10<07:26,  1.22it/s]
 60%|██████    | 15/25 [00:13<00:07,  1.36it/s]
Deforum progress:   3%|▎         | 15/556 [00:11<06:34,  1.37it/s]
 68%|██████▊   | 17/25 [00:15<00:05,  1.45it/s]
Deforum progress:   3%|▎         | 17/556 [00:12<06:09,  1.46it/s]
 76%|███████▌  | 19/25 [00:16<00:03,  1.51it/s]
Deforum progress:   3%|▎         | 19/556 [00:13<05:54,  1.51it/s]
 84%|████████▍ | 21/25 [00:17<00:02,  1.53it/s]
Deforum progress:   4%|▍         | 21/556 [00:15<05:49,  1.53it/s]
 92%|█████████▏| 23/25 [00:18<00:01,  1.55it/s]
Deforum progress:   4%|▍         | 23/556 [00:16<05:44,  1.55it/s]
100%|██████████| 25/25 [00:19<00:00,  1.26it/s]

Deforum progress:   4%|▍         | 25/556 [00:17<05:41,  1.56it/s]Animation frame: 2/120  
Creating in-between cadence frame: 0; tween:0.50;
Deforum progress:   4%|▍         | 25/556 [00:19<06:50,  1.29it/s]
Error completing request
Arguments: ('task(5mlekxrx0qfvrqa)', None, False, None, '3D', 120, 'replicate', '0:(0)', '0:(1.0025+0.002*sin(1.25*3.14*t/30))', '0:(0)', '0:(0)', '0:(1.75)', '0:(0.5)', '0:(0.5)', '0:(0)', '0:(0)', '0:(0)', False, '0:(0)', '0:(0)', '0:(0)', '0:(53)', '0: (0.065)', '0: (0.65)', '0: (1.0)', '0: (7)', '0:(1.5)', False, '0:(1)', '0:(0)', False, '0: (25)', '0: (70)', '0: (1)', False, '0: (200)', '0: (10000)', '0:(s), 1:(-1), "max_f-2":(-1), "max_f-1":(s)', False, '0: ("Euler a")', '0: ("{video_mask}")', False, '0: ("{video_mask}")', False, '0: ("model1.ckpt"), 100: ("model2.safetensors")', False, '0: (2)', True, '0: (1.05)', '0: (5)', '0: (1.0)', '0: (0.1)', '0: (0.0)', 'LAB', '', 1.0, False, '2', 'None', '0: (1)', 'None', '0: (1)', '0', 'perlin', 8, 8, 4, 0.5, True, False, 0.2, 'border', 'bicubic', False, 'https://deforum.github.io/a1/V1.mp4', 1, 0, -1, False, False, 'https://deforum.github.io/a1/VM1.mp4', False, '20230129210106', '{\n    "0": "tiny cute swamp bunny, highly detailed, intricate, ultra hd, sharp photo, crepuscular rays, in focus, by tomasz alen kopera",\n    "30": "anthropomorphic clean cat, surrounded by fractals, epic angle and pose, symmetrical, 3d, depth of field, ruan jia and fenghua zhong",\n    "60": "a beautiful coconut --neg photo, realistic",\n    "90": "a beautiful durian, trending on Artstation"\n}\n    ', '', '', 512, 512, False, False, -1, 'Euler a', False, 0, 0, 25, 0.0, 1, True, True, False, False, False, 'Deforum_{timestring}', '{timestring}_{index}_{prompt}.png', 'iter', 1, False, False, True, 0.8, 'https://deforum.github.io/a1/I1.png', False, False, False, True, 'https://deforum.github.io/a1/M1.jpg', 1.0, 1.0, 4, 1, True, 4, 'reroll', 10.0, False, 15, False, False, 'FFMPEG mp4', 'None', 'https://deforum.github.io/a1/A1.mp3', False, 'realesr-animevideov3', 'x2', True, False, 'x0_pred', 'C:/SD/20230124234916_%09d.png', 'testvidmanualsettings.mp4', False, 'None', 2, False, 2, False, '', True, False, 'None', True, 'None', False, 'RAFT', 'None', False, 'None', False, 'None', False, False, '0:(0.5)', '0:(1)', '0:(0.5)', '0:(1)', '0:(100)', '0:(0)', False, '{\n    "0": "https://deforum.github.io/a1/Gi1.png",\n    "max_f/4-5": "https://deforum.github.io/a1/Gi2.png",\n    "max_f/2-10": "https://deforum.github.io/a1/Gi3.png",\n    "3*max_f/4-15": "https://deforum.github.io/a1/Gi4.jpg",\n    "max_f-20": "https://deforum.github.io/a1/Gi1.png"\n}', '0:(0.75)', '0:(0.35)', '0:(0.25)', '0:(20)', '0:(0.075)', None, None, True, '', '', False, False, False, 'none', 'None', 1, 0, 1, 64, 64, 64, 'Inner Fit (Scale to Fit)', None, None, True, '', '', False, False, False, 'none', 'None', 1, 0, 1, 64, 64, 64, 'Inner Fit (Scale to Fit)', None, None, True, '', '', False, False, False, 'none', 'None', 1, 0, 1, 64, 64, 64, 'Inner Fit (Scale to Fit)', None, None, True, '', '', False, False, False, 'none', 'None', 1, 0, 1, 64, 64, 64, 'Inner Fit (Scale to Fit)', None, None, True, '', '', False, False, False, 'none', 'None', 1, 0, 1, 64, 64, 64, 'Inner Fit (Scale to Fit)') {}
Traceback (most recent call last):
  File "/Users/username/Documents/GitHub/brkirch/stable-diffusion-webui/modules/call_queue.py", line 56, in f
    res = list(func(*args, **kwargs))
  File "/Users/username/Documents/GitHub/brkirch/stable-diffusion-webui/modules/call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "/Users/username/Documents/GitHub/brkirch/stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts/deforum.py", line 101, in run_deforum
    render_animation(args, anim_args, video_args, parseq_args, loop_args, controlnet_args, root.animation_prompts, root)
  File "/Users/username/Documents/GitHub/brkirch/stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts/deforum_helpers/render.py", line 305, in render_animation
    depth = depth_model.predict(turbo_next_image, anim_args.midas_weight, root.half_precision)
  File "/Users/username/Documents/GitHub/brkirch/stable-diffusion-webui/extensions/deforum-for-automatic1111-webui/scripts/deforum_helpers/depth.py", line 107, in predict
    midas_depth = torch.nn.functional.interpolate(
  File "/Users/username/Documents/GitHub/brkirch/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/functional.py", line 3967, in interpolate
    return torch._C._nn.upsample_bicubic2d(input, output_size, align_corners, scale_factors)
RuntimeError: "compute_indices_weights_cubic" not implemented for 'Half'

Traceback (most recent call last):
  File "/Users/username/Documents/GitHub/brkirch/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/routes.py", line 337, in run_predict
    output = await app.get_blocks().process_api(
  File "/Users/username/Documents/GitHub/brkirch/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1018, in process_api
    data = self.postprocess_data(fn_index, result["prediction"], state)
  File "/Users/username/Documents/GitHub/brkirch/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/blocks.py", line 935, in postprocess_data
    if predictions[i] is components._Keywords.FINISHED_ITERATING:
IndexError: tuple index out of range

Additional information

I tried to fix this myself to the best of my current knowledge: For a temp fix I had to add these lines in Deforum's deforum-for-automatic1111-webui/scripts/deforum_helpers/depth.py file, before calling to midas_depth = torch.nn.functional.interpolate() (line 103):

if self.device.type == "mps" and half_precision:
            midas_depth = midas_depth.to(torch.float32)
            img_midas = np.float32(img_midas)

I realise this is in Deforum's repo, but it happens only on Mac OS, with or without --upcast-sampling, not to mention --no-half doesn't work at all and throws an error when starting generation, but that's a different issue. Deforum group don't have any Mac developers so I thought I'll lend a hand