Closed bshikin closed 1 year ago
Add TestMultiprocessLocalLLM, a class that can wrap an LLM implementation and run inference (aka query method) in separate processes on the local box.
TestMultiprocessLocalLLM
query
The configuration would look like:
llm: class: MultiprocessLocalLLM llm_config: class: <wrapped LLM class> config <wrapped LLM config> wrapper_config num_workers: 5, query_timeout_sec: 10
Add
TestMultiprocessLocalLLM
, a class that can wrap an LLM implementation and run inference (akaquery
method) in separate processes on the local box.The configuration would look like: