davmacario / MDI-LLM

Implementation of Model-Distributed Inference for Large Language Models, built on top of LitGPT
MIT License
3 stars 2 forks source link

Add prompt support #25

Closed davmacario closed 6 months ago

davmacario commented 6 months ago

Tested - successful. Closes #12

Prompt can be provided either:

Also allows to generate of fewer samples than n. of nodes, and raise a warning in this case.