search

invoke-ai / InvokeAI

InvokeAI is a leading creative engine for Stable Diffusion models, empowering professionals, artists, and enthusiasts to generate and create visual media using the latest AI-driven technologies. The solution offers an industry leading WebUI, supports terminal use through a CLI, and serves as the foundation for multiple commercial products.
https://invoke-ai.github.io/InvokeAI/
Apache License 2.0
22.78k stars 2.35k forks source link

Run image generation and upscaling in parallel #862

Closed whosawhatsis closed 1 year ago

whosawhatsis commented 1 year ago

When generating large sets of images, my GPU is pegged while generating images (txt2img), but my CPU is idle. When that finishes, it starts running Real-ESRGAN and GFPGAN on all of those images, which uses close to half of my CPU time and none of my GPU for nearly as long.

This is on an M1 Max, and the optimum might be different when using CUDA, but it would be nice to have an option to run the upscaling for one image in parallel with generating the next image, rather than doing all of the generation, then all of the upscaling after that finishes.

A related issue is that, if I decide to stop the image generation early, nothing gets upscaled. This would also be fixed by my suggestion above. Even if it instead alternated, generating one image, then upscaling it, then generating the next image, and so on, ending generation early could still result in upscaled images. This would result in alternating between CPU and GPU tasks, which might if nothing else, at least have the benefit of reducing the maximum temperature (and thus fan noise level) of the computer during the whole process.

timdesrochers commented 1 year ago

I also find it a little frustrating when I decide to cut a batch generation short and therefore kill it before GFPGAN/ESRGAN hits any of the gens.

blessedcoolant commented 1 year ago

We implemented GFPGAN and ESRGAN very early. During the initial implementation, the post processing effects ran right after image generation. But the issue with this was that they obviously took more memory and as a result the performance of the image generation was itself taking a huge hit .. and in a lot of cases just outright crashing due to memory issues -- using 8GB cards as the standard because that's what a large chunk of people have.

And given most people generate 1 image at a time, we decided the work around here was to run them AFTER the image generation so we can avoid the crashes. This obviously meant that multiple image generations would have all the images processed after the completion of the image generation because the image generation loop wouldn't end before that.

But obviously a LOT has changed since then. The memory optimization work that has been done has greatly lowered memory footprint of the image generation process. Maybe it is now a viable strategy once again to look at post processing effects running right after image generation.

I'll take a look at this over the weekend and see if there's a way to implement this without a switch. But if not, I guess we can still add a switch for users who have systems that are capable of handling this.

psychedelicious commented 1 year ago

I'll note that in the web UI, the sequence is as you request - generation1, upscale1, facefix1, generation2, upscale2, facefix2, and so on.

If you cancel before an image has gone through request postprocessing, though, it is not saved. I suppose it should save it if you cancel after generation but before post?

whosawhatsis commented 1 year ago

Are you sure? I just got the webui running and tried it. I set it to generate 10 images. It did that, then began upscaling them. Maybe you're running an older version or a different branch? I just installed mine two days ago...

psychedelicious commented 1 year ago

I suppose you are on the main branch? On development, there is a brand new UI which works as I described (that's how it should work, anyways).

whosawhatsis commented 1 year ago

Ah, cool, I'll check that out.

whosawhatsis commented 1 year ago

Yes, the development version of the webui does do the upscaling for each image, which I like a lot better. I still think it would be useful to be able to do that in parallel with generating the next image, though, since (at least on apple silicon), one uses the CPU and the other uses the GPU, so they don't step on each other much.

psychedelicious commented 1 year ago

@Kyle0654 With the new backend, will the workers process jobs in parallel e.g. iteration 1's postprocessing can happen in parallel with interation 2's generation?

Kyle0654 commented 1 year ago

@Kyle0654 With the new backend, will the workers process jobs in parallel e.g. iteration 1's postprocessing can happen in parallel with interation 2's generation?

It could be done, but it's a challenging problem with the new node-based stuff, since I can't readily tell what resources would need to be locked to process a node, and those could change based on settings.