Multi GPU

[St33lMouse]

Feature Idea

Allow memory to split across GPUs. With the arrival of Flux, even 24gb cards are maxed out and models have to be swapped in and out in the image creation process, which is slow. If you have two gpus this would be a massive speed increase.

Existing Solutions

No response

Other

No response

[BrechtCorbeel]

I have 2 GPUS one one of my pcs (1070 + 3060) and the terminal does recognise both of them, I have not yet seen functionality to pick which one to use though, nor have I ran my dual GPU setup and checked whether it utilises both, maybe it does by default?

Torch after googling a bit does as I suspected have capabilities to run multi GPU nodes.

The schnell blew all resolution out of proportion doing ridiculous resolutions really fast.

For me personally, what is coming out of my 4090 is larger then my midjourney files even without upscaling and upscaling is not that big of a deal.

I don't really know if it would even work or how much of a bottleneck crosstalk on gpus is calculation wise.

I know or think I know I have seen or heard on automatic1111 or wherever it was, there is functionality to do tiles per gpu and run multiple that way.

I do think going to real time 4 or 8K video models yes there 80 or 160 GB Vram crads are going to be a big benefit.

Anyone wants to start a GPU dedicated computer bussiness that sells AI specific design industrial computers for dev use with me I think this would be a great business to start, the market for it is booming, I don't think we are far off of building server tower level and size computers for at home use for enthusiasts.

[ltdrdata]

I have 2 GPUS one one of my pcs (1070 + 3060) and the terminal does recognise both of them, I have not yet seen functionality to pick which one to use though, nor have I ran my dual GPU setup and checked whether it utilises both, maybe it does by default?

Torch after googling a bit does as I suspected have capabilities to run multi GPU nodes.

The schnell blew all resolution out of proportion doing ridiculous resolutions really fast.

For me personally, what is coming out of my 4090 is larger then my midjourney files even without upscaling and upscaling is not that big of a deal.

I don't really know if it would even work or how much of a bottleneck crosstalk on gpus is calculation wise.

I know or think I know I have seen or heard on automatic1111 or wherever it was, there is functionality to do tiles per gpu and run multiple that way.

I do think going to real time 4 or 8K video models yes there 80 or 160 GB Vram crads are going to be a big benefit.

Anyone wants to start a GPU dedicated computer bussiness that sells AI specific design industrial computers for dev use with me I think this would be a great business to start, the market for it is booming, I don't think we are far off of building server tower level and size computers for at home use for enthusiasts.

The problem with dividing into tiles is that context is lost between tiles, which results in a decrease in the overall consistency of the image.

[BrechtCorbeel]

I have 2 GPUS one one of my pcs (1070 + 3060) and the terminal does recognise both of them, I have not yet seen functionality to pick which one to use though, nor have I ran my dual GPU setup and checked whether it utilises both, maybe it does by default? Torch after googling a bit does as I suspected have capabilities to run multi GPU nodes. The schnell blew all resolution out of proportion doing ridiculous resolutions really fast. For me personally, what is coming out of my 4090 is larger then my midjourney files even without upscaling and upscaling is not that big of a deal. I don't really know if it would even work or how much of a bottleneck crosstalk on gpus is calculation wise. I know or think I know I have seen or heard on automatic1111 or wherever it was, there is functionality to do tiles per gpu and run multiple that way. I do think going to real time 4 or 8K video models yes there 80 or 160 GB Vram crads are going to be a big benefit. Anyone wants to start a GPU dedicated computer bussiness that sells AI specific design industrial computers for dev use with me I think this would be a great business to start, the market for it is booming, I don't think we are far off of building server tower level and size computers for at home use for enthusiasts.

The problem with dividing into tiles is that context is lost between tiles, which results in a decrease in the overall consistency of the image.

I think it had a method of doing a really low level overall larger context tile, of the entire image and then it generated tiles with overlap that was very consistent.

It was part of an image upscaler method in Automatic1111

the only problem was that then you had an absurdly highly detailed image all over the place if it was overdone with too many steps (meaning micro characters etc. starting to apear and not doing textures but the prompt in the tile)

[St33lMouse]

You're in luck. This will knock 30 seconds off your inference time:

https://www.reddit.com/r/StableDiffusion/comments/1el79h3/flux_can_be_run_on_a_multigpu_configuration/

[jdc4429]

The multi GPU workflow is pretty cool but you can't select which gpu to use for the main model... I asked if this could be implemented and he said that he can't because of the way comfyUI implements it and that would have to be changed. I was thinking if it would be possible to create something on top of the existing code like an extension or node that allows you to select which gpu is used regarding the main model... Maybe an advanced version of load checkpoint that passes this data for which gpu to use? For the multi gpu workflow to work you can't specify which gpu to use at the startup otherwise the drop down menu's will not have one of the gpu options. There should be a way to do this that's not that difficult.

[alcarinque-istar]

These nodes allow to chose in which gpu the model is loaded https://github.com/neuratech-ai/ComfyUI-MultiGPU

[jdc4429]

Excellent! Thank you!

On Wed, Aug 7, 2024 at 8:09 PM alcarinque-istar @.***> wrote:

These nodes allow to chose in which gpu the model is loaded https://github.com/neuratech-ai/ComfyUI-MultiGPU

— Reply to this email directly, view it on GitHub https://github.com/comfyanonymous/ComfyUI/issues/4238#issuecomment-2274574167, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEVP2REAGTNE5CUY543PAMTZQKZJ5AVCNFSM6AAAAABMBUTAPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZUGU3TIMJWG4 . You are receiving this because you commented.Message ID: @.***>

[yincangshiwei]

https://github.com/mit-han-lab/distrifuser

This effect seems great. I hope ComfyUI can be integrated and tried.

[Lay2000]

https://github.com/xdit-project/xDiT This is a great project to support DiT-typed(Stable Diffusion3, Flux, HunyuanDiT) models executing on multiple GPUs and can achieve remarkable speed-up without any visible quality loss. I hope we can collaborate to integrate ComfyUI and xDiT together.