Add support for generating > n_nodes samples

davmacario / MDI-LLM

Implementation of Model-Distributed Inference for Large Language Models, built on top of LitGPT

MIT License

3 stars 2 forks source link

Add support for generating > n_nodes samples #10

Closed davmacario closed 6 months ago

davmacario commented 7 months ago

This should reduce the idle time of the devices (as we would be adding more computation). It would also allow to compare the 2-node and 3-node cases over the same amount of generated tokens.