search

ethansmith2000 / comfy-todo

MIT License
68 stars 4 forks source link

Can you write an extension for Forge or A1111 ? #1

Open wardensc2 opened 7 months ago

wardensc2 commented 7 months ago

Hi ethan

Can you support us write an extension for A1111 or Forge. Your plugin not only improve render speed, it also reduce about 30% or more Vram when upscale and really necessary for us to create high resolution image

Thank you so much

ethansmith2000 commented 7 months ago

Hi there, I'm currently looking into it, the closest I see is that ToMe is applied here although its not clear where it imports from.

https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/dev/modules/sd_models.py#L866

wardensc2 commented 7 months ago

I think that is token merge in Cross attention optimization in Settings but it can't get much speed or reduce Vram like your node, I already try but the result is bad, the image is uglier

image

wardensc2 commented 7 months ago

People already update your node into Forge but still no one write an extension for A1111 yet sadly

AG-w commented 7 months ago

forge added model patcher similar to comfyUI, can't just copy them directly

AG-w commented 7 months ago

People already update your node into Forge but still no one write an extension for A1111 yet sadly

I'm not sure if it's work 100% correctly, but you can try https://github.com/AG-w/sd-webui-todo/

wardensc2 commented 7 months ago

People already update your node into Forge but still no one write an extension for A1111 yet sadly

I'm not sure if it's work 100% correctly, but you can try https://github.com/AG-w/sd-webui-todo/

Thank you so much

ethansmith2000 commented 7 months ago

we're also exploring it over here https://github.com/ethansmith2000/ImprovedTokenMerge/issues/2 the current method im using lets us circumvent having to use the setup a1111 uses for ToMe but also still unsure on some other parts

wardensc2 commented 7 months ago

People already update your node into Forge but still no one write an extension for A1111 yet sadly

I'm not sure if it's work 100% correctly, but you can try https://github.com/AG-w/sd-webui-todo/

Hi AG-w I try to run it on A1111 so far it only work in some resolution, I try to render 1920x1080 A1111 will stop running even when I disable this extension. Thank for your effort it seem that we need to wait for ethansmith2000 to solve this problem

feffy380 commented 7 months ago

I was implementing ToDo for kohya/sd-scripts (I found it greatly speeds up training). The code was self-contained enough that I was able to turn it into an extension. https://github.com/feffy380/sd-webui-token-downsampling

Instead of individual downsample factors for different depths, I've opted to add a single max_depth slider which determines which layers are affected (basically tomesd's max_downsample argument but renamed to avoid confusion). I found the quality/speed tradeoff terrible at deeper layers so I didn't feel the granularity was necessary.

There's also the option to disable ToDo after a certain percentage of steps. I found this helps with details, though you sacrifice some speed. You're essentially using the original model as a refiner for the ToDo model.

ethansmith2000 commented 7 months ago

Awesome to hear! The reason why I set multiple depths is because for 2048x2048 I found using 4 and 2 for the top layer and one below it respectively works pretty well.

feffy380 commented 7 months ago

Ah fair enough, I hadn't gone that high. I'll see about adding sliders to override the downsample factor for individual layers.