using this method I was able to perform inference against local models hosted by Jan. Unfortunately my TensorRT Mistral model had streaming issues but I got other models working partially. .streamlit/secrets.toml does seem to need a valid openai API key, im not seeing any usage, but worth noting.
modified core/builder_config.py ---
from llama_index.llms.openai_like import OpenAILike
API_KEY = os.getenv('OPENAI_API_KEY') BUILDER_LLM = OpenAILike( api_base="[IP]:1337", model="[model ID]", is_chat_model=True, max_tokens=None, api_version="v1", api_key=API_KEY, )
using this method I was able to perform inference against local models hosted by Jan. Unfortunately my TensorRT Mistral model had streaming issues but I got other models working partially. .streamlit/secrets.toml does seem to need a valid openai API key, im not seeing any usage, but worth noting.