Open acruikshank-wm opened 11 months ago
Interaction.tsx
's runOnce
function is hardwired to make an api call vs. having the option to interact with llm's via alternative methods (e.g., via ggm.bin file or via huggingface hub using ctransformers). An obvious way to conditionally interact with an llm is to provide the function with an llm type. In this scenario, the default interaction can be to call an openai api. If the llm type is huggingface hub, then it can call a new function that uses ctransformers.
Description
@vince-westmonroe has been playing around with locally hosted LLMs and LangChain. It turns out that LangChain has internal support for the CTransformers package that will download models from HuggingFaceHub and run them locally. This includes the quantized Llama-2 models that are SOTA for open source. It would be really nice to have this available in TestBench.
Acceptance