Local Stable Diffusion Render

[jasonmhead]

Describe the feature you'd like to see:

Any plans to support generating local diffusion images locally? Perhaps it could tie into Automatic1111's API (with it being a dependency to install) after launching it and save a lot of work that way, and just pull it into Blender from there.

Additional information

No response

[Norgus]

OK My hunch must have been correct but my solution of deleting all the scripts was too drastic - I restored just the ones from the checkpoint so the scripts folder had the default extras but no more and AI-Render works!

I can't wait for there to be a proper API so this nonsense doesn't happen in the future. Sorry for the headache @benrugg

[benrugg]

Hey, that's great! I'm so glad you finally got it working. Love that default sugar cube 😄

[Norgus]

donut tutorial cheating speedrun

[benrugg]

Hahaha love it

[jamesAllenJahner3rd]

@Jitone1, Blender has a really dumb thing, where it doesn't actually activate a newly installed add-on if you had an older version already running. Restart Blender and you should see a new box in the add-on preferences where you can activate local SD.

@benrugg Hey yeah, I updated everything reloaded blender and it's working. Thanks. So basically this seems to be doing image to image using the original render as the input?

[enzyme69]

Testing "Local Render", I have StableDiffusionWebUI, but getting error message in the log:

Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Error completing request
Arguments: ('0', 'beautiful portrait, 8k resolution concept art portrait, hyperdetailed, intricately detailed, trending on Artstation, triadic colors, volumetric lighting, soft focus, dynamic lighting', '', 'None') {}
Traceback (most recent call last):
  File "/Users/blendersushi/Documents/stable-diffusion-webui/modules/ui.py", line 223, in f
    res = list(func(*args, **kwargs))
  File "/Users/blendersushi/Documents/stable-diffusion-webui/modules/ui.py", line 148, in save_files
    p = MyObject(data)
  File "/Users/blendersushi/Documents/stable-diffusion-webui/modules/ui.py", line 143, in __init__
    for key, value in d.items():
AttributeError: 'int' object has no attribute 'items'

[enzyme69]

I saw this and it might do something... https://github.com/benrugg/AI-Render/issues/2#issuecomment-1287802853

Still waiting because it might take like 4 minutes in my system.....

zsh: suspended  ./run_webui_mac.sh
(base) blendersushi@192-168-1-101 stable-diffusion-webui % 
(base) blendersushi@192-168-1-101 stable-diffusion-webui % 
(base) blendersushi@192-168-1-101 stable-diffusion-webui % 
(base) blendersushi@192-168-1-101 stable-diffusion-webui % git pull
Already up to date.
(base) blendersushi@192-168-1-101 stable-diffusion-webui % git checkout f49c08e
Note: switching to 'f49c08e'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at f49c08e prevent error spam when processing images without txt files for captions
(base) blendersushi@192-168-1-101 stable-diffusion-webui % ./run_webui_mac.sh  
To make your changes take effect please reactivate your environment
WARNING: overwriting environment variables set in the machine
overwriting variable {'PYTORCH_ENABLE_MPS_FALLBACK'}
You are not currently on a branch.
Please specify which branch you want to rebase against.
See git-pull(1) for details.

    git pull <remote> <branch>

WARNING:root:Pytorch pre-release version 1.13.0.dev20220922 - assuming intent to test it
Warning: caught exception 'Torch not compiled with CUDA enabled', memory monitor disabled
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Loading weights [81761151] from /Users/blendersushi/Documents/stable-diffusion-webui/models/Stable-diffusion/v1-5-pruned-emaonly.ckpt
Global Step: 840000
Applying v1 cross attention optimization.
Model loaded.
Loaded a total of 0 textual inversion embeddings.
Embeddings: 
Running on local URL:  http://127.0.0.1:7861

To create a public link, set `share=True` in `launch()`.

NOTE: I also have Swift Diffusion -> on Mac M1 8GB -> slow, but it's running at 1 minute for 25 samples. Maybe if somewhat we could hook this AI Render "Local Diffusion" somewhat into Swift Diffusion (using bazel command etc).

[enzyme69]

Still error-ing:

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/web-ui/lib/python3.10/site-packages/gradio/routes.py", line 273, in run_predict
    output = await app.blocks.process_api(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/web-ui/lib/python3.10/site-packages/gradio/blocks.py", line 739, in process_api
    inputs = self.preprocess_data(fn_index, inputs, state)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/web-ui/lib/python3.10/site-packages/gradio/blocks.py", line 638, in preprocess_data
    processed_input.append(block.preprocess(raw_input[i]))
  File "/opt/homebrew/Caskroom/miniforge/base/envs/web-ui/lib/python3.10/site-packages/gradio/components.py", line 1067, in preprocess
    return self.choices.index(x)
ValueError: True is not in list

[benrugg]

@enzyme69 glad you got it running by checking out the specific commit. If you got an error later, I would guess that maybe it got reset back to a later commit, so I would try doing git checkout f49c08e again. And yeah, it's really slow on my M1 mac, too. If Swift Diffusion ends up getting further down the road, let me know and I'll try to integrate with it!

@Jitone1, yep exactly

[enzyme69]

@benrugg Still working on it, maybe it's working ....

Try this Swift Diffusion: and see if that can work somewhat. 512 x 512 == under 1 minute with 25 samples. https://forums.swift.org/t/started-to-work-on-swift-diffusion-a-port-of-stable-diffusion-in-swift/60302

Requires Bazel etc, but amazing speed and no pressure on the system.

[enzyme69]

@benrugg Still failing with same error, I will reinstall using latest addon:

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/web-ui/lib/python3.10/site-packages/gradio/routes.py", line 273, in run_predict
    output = await app.blocks.process_api(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/web-ui/lib/python3.10/site-packages/gradio/blocks.py", line 739, in process_api
    inputs = self.preprocess_data(fn_index, inputs, state)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/web-ui/lib/python3.10/site-packages/gradio/blocks.py", line 638, in preprocess_data
    processed_input.append(block.preprocess(raw_input[i]))
  File "/opt/homebrew/Caskroom/miniforge/base/envs/web-ui/lib/python3.10/site-packages/gradio/components.py", line 1067, in preprocess
    return self.choices.index(x)
ValueError: 0 is not in list

I am deleting StableDiffusionWebUI and start from scratch. Maybe I might got lucky.

[enzyme69]

No luck, same old error message:

Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Error completing request
Arguments: ('0', 'three baboon heads on top of each other on a plate, beautiful portrait, 8k resolution concept art portrait, hyperdetailed, intricately detailed, trending on Artstation, triadic colors, volumetric lighting, soft focus, dynamic lighting', '', 'None') {}
Traceback (most recent call last):
  File "/Users/blendersushi/Documents/stable-diffusion-webui-master/stable-diffusion-webui/modules/ui.py", line 223, in f
    res = list(func(*args, **kwargs))
  File "/Users/blendersushi/Documents/stable-diffusion-webui-master/stable-diffusion-webui/modules/ui.py", line 148, in save_files
    p = MyObject(data)
  File "/Users/blendersushi/Documents/stable-diffusion-webui-master/stable-diffusion-webui/modules/ui.py", line 143, in __init__
    for key, value in d.items():
AttributeError: 'int' object has no attribute 'items'

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/web-ui/lib/python3.10/site-packages/gradio/routes.py", line 275, in run_predict
    output = await app.blocks.process_api(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/web-ui/lib/python3.10/site-packages/gradio/blocks.py", line 791, in process_api
    predictions = self.postprocess_data(fn_index, result["prediction"], state)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/web-ui/lib/python3.10/site-packages/gradio/blocks.py", line 742, in postprocess_data
    if predictions[i] is components._Keywords.FINISHED_ITERATING:
IndexError: tuple index out of range

[enzyme69]

I don't have any environment would this matter? @Norgus how did you fix yours?

[enzyme69]

git checkout f49c08e

Okey... the address change to http://127.0.0.1:7861

This seems to work and running

[Wanglibin14715]

Hi, I run AI-Render-v0-3-2.zip add on, it partially work. I can see the result in my webui/output file, however it seem to have problem returning result to blender. Blender say Couldn't load the image from SD. Then, I tried AI-Render-v0-3-4.zip, same problem.

[benrugg]

@enzyme69, glad you got it working. Yeah, at least until I can make a more stable integration, git checkout f49c08e is crucial.

@Wanglibin14715, definitely use v0.3.4, and try all the steps here: https://github.com/benrugg/AI-Render/wiki/Local-Installation

[benrugg]

Hi everyone - I just made a new release that should work much more reliably with Automatic1111 going forward. You'll need to follow slightly updated instructions (the key difference being that you will now enable API mode): https://github.com/benrugg/AI-Render/wiki/Local-Installation

I've tested on Windows and Mac, but if a couple of you could verify that it works for you, that would be amazing. I will release this publicly as soon as I know it works!

[Norgus]

@benrugg I'm about to update things and test but just a heads-up, your instrcutions on that page, point 5 say

Run notepad.exe .\webui.bat

which should be webui-user.bat

[Norgus]

I can confirm that it worked first time on up-to-date automatic1111 SD repository!

[benrugg]

@Norgus, yes, thank you so much for catching that! And for testing so quickly! Does it happen work with your other user scripts now?

[Norgus]

@benrugg very welcome, thanks for making the cool addon & updating to support local installs!

I had some subset of my scripts loaded there, but I just copied across the rest, restarted SD, and the API continues to do its job!

[benrugg]

@Norgus great. So glad it's working well now 👍

[enzyme69]

I use the latest addon, follow all instruction for Mac to add the --api, but getting error: AI Render Error: It looks like the Automatic1111 server is running, but it's not in API mode

Which is annoying because I did run it in API mode?

[enzyme69]

Okey seems like I still need to do this everytime: git checkout df0a1f8

[Wanglibin14715]

@enzyme69, glad you got it working. Yeah, at least until I can make a more stable integration, git checkout f49c08e is crucial.

@Wanglibin14715, definitely use v0.3.4, and try all the steps here: https://github.com/benrugg/AI-Render/wiki/Local-Installation

Same problem, Friend. "Couldn't load the image from SD." Do you know what I might have done leading to myblender cannot load image, I didn't find out I had done some thing special, it's weird.

[benrugg]

@enzyme69 I will check into whether something broke after df0a1f8. Glad it at least works on that commit.

@Wanglibin14715 can you confirm that you're using v0.4.0 and following the new instructions for api mode?

[benrugg]

Closing this now - implemented in 433637e39bfd10fb39ed1489e1bf4266d75c2067