The goal of this project is to allow users to easily load their locally hosted language models in a notebook for testing with Langchain. There are currently three notebooks available. Two of them use an API to create a custom Langchain LLM wrapper—one for oobabooga's text generation web UI and the other for KoboldAI. The third notebook loads the models without an API by leveraging the oobabooga's text-generation-webui virtual environment and modules for model loading.
You will end up with an instance of the Custom LLM Wrapper that can be used to generate text:
llm("prompt goes here")
You can use this instead of the OpenAI LLM class that you see used in most of the guides and documentation.
Please follow the setup instructions for the APIs provided in their respective repositories. Just update the url variable with your api url then run the cells to create an instance of the Custom LLM Wrapper.
Using the API is now my preferred method for loading the models. I plan on improving the API classes/notebooks, but for now, they work quite well. I'm leaving the non-api stuff up for now, but I won't be actively maintaining them in the future, so things might break.
pip install jupyter
in your preferred command prompt or terminal.jupyter notebook
in the command prompt or terminal to launch the Jupyter interface.Non-API-Notebook.ipynb
is located (ooba users put it in ./text-generation-webui/
and open the notebook in the Jupyter interface.