davmacario / MDI-LLM

Implementation of Model-Distributed Inference for Large Language Models, built on top of LitGPT
MIT License
3 stars 2 forks source link

Add prompt support for MDI #12

Closed davmacario closed 6 months ago

davmacario commented 7 months ago

Idea: use a list of prompts, encode them as in sample.py, and allow them to be different to generate n_nodes different samples.

Need to figure out how to input them; my idea would be to provide a text file, and each line is one prompt; extra lines are either ignored, or, if #10 is completed, are considered as well.