Open avianion opened 1 month ago
Can you shard the model across 8 gpus in a single node? I'm having trouble even using a single node.
Hi, @avianion have you solve this issue? I have the same issue for llama405b across nodes inference on HPC cluster.
Hello
We are trying to use Deepspeed to load LLama 405b across 2 nodes, of 8 x H100 SXM each.
We want to shard the model across all 16 gpus so that the model will be loaded in shards of 50.6GB each and then run inference and training with the model.
Please provide a guide on exactly how to do this, as we have been unable to figure it out.