search

VectorSpaceLab / OmniGen

OmniGen: Unified Image Generation. https://arxiv.org/pdf/2409.11340
MIT License
2.88k stars 227 forks source link

Multi gpu feasible? #79

Open matbee-eth opened 3 weeks ago

matbee-eth commented 3 weeks ago

I have 2x 4090, is it possible to have this model run across multiple gpus, or offload the kv cache to another cuda device? I'm fairly certain its the kv_cache being offloaded to cpu thats causing it to be so slow.

staoxiao commented 3 weeks ago

Sorry, the current code doesn't support multi-gpus. We will optimize these functions in the future. For inference time, you can refer to https://github.com/VectorSpaceLab/OmniGen/blob/main/docs/inference.md#requiremented-resources

waiwong614 commented 3 weeks ago

Oh bummer, I have 2x 1070, and just realized it only using one GPU. Keep up the good work and hope to see more optimization and can use low VRAM GPU mode.

HarjotSingh-b18055 commented 2 weeks ago

Sorry, the current code doesn't support multi-gpus. We will optimize these functions in the future. For inference time, you can refer to https://github.com/VectorSpaceLab/OmniGen/blob/main/docs/inference.md#requiremented-resources

Hey, @staoxiao will this thing make it work?

https://stackoverflow.com/questions/54216920/how-to-use-multiple-gpus-in-pytorch