Question: How to run multi-node inference?

bigscience-workshop / Megatron-DeepSpeed

Ongoing research training transformer language models at scale, including: BERT & GPT-2

Other

1.31k stars 213 forks source link

Question: How to run multi-node inference? #317

Closed asaparov closed 2 years ago

asaparov commented 2 years ago

I was wondering how to perform multi-node inference? I'm very interested in running some experiments on Bloom (specifically probing its ability to reason). The high-level descriptions of Zero and DeepSpeed Inference indicate that multi-node inference is supported, but the examples I've found so far are only of multi-node training.

I really appreciate any tips or pointers!

asaparov commented 2 years ago

I was able to get deepspeed to perform inference instead of training in the multi-node setting on SLURM, but I ran into a different error. I'll open a different issue for that (#318).