multi-gpu support?

[bluenevus]

I see the container running and I attached 2 gpus to each container however, it only loads in one gpu vs spreading across the 2 gpus.

that is when I look at nvtop, I can see that its only using onegpu vs spreading across all gpus and filling that vram and only one gpu is actually used when testing at command line

I'm not talking gradio because I see the issue someone else posted and saw the wont fix

[zhengzangw]

Did you follow the instructions here, where you should launch the inference with torchrun.

We tested that the sequence-parallelism is working.