Working with open source models

[zbabarMRMR]

Is it possible to work with Open source models like LLama2 using Ollama? I have tried but somehow it did not work and constantly asks for OpenAI key.

[xn3cr0nx]

It depends on the framework you are using. For instance, using LlamaIndex, it's not enough to setup your LLM, you will also need to setup an embedding model to avoid LlamaIndex to rely on OpenAI by default.

For instance, using Ollama, and Qdrant as vector storage, you can implement something like this:

import qdrant_client

from llama_index import VectorStoreIndex, Document, ServiceContext, StorageContext, set_global_handler, SimpleDirectoryReader
from llama_index.llms import Ollama
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.embeddings import FastEmbedEmbedding

set_global_handler("simple")

client = qdrant_client.QdrantClient(path="./qdrant")
vector_store = QdrantVectorStore(client=client, collection_name="testing")
storage_context = StorageContext.from_defaults(vector_store=vector_store)

llm = Ollama(model="llama2", request_timeout=300.0, streaming=True)
embed_model = FastEmbedEmbedding(model_name="BAAI/bge-small-en-v1.5")

service_context = ServiceContext.from_defaults(llm=llm, embed_model=embed_model)

index = VectorStoreIndex([], service_context=service_context, storage_context=storage_context)

query_engine = index.as_query_engine(service_context=service_context, streaming=True)

# create documents
doc1 = Document(text="Molly is a cat")
doc2 = Document(text="Doug is a dog")

# add to the index
index.insert(doc1)
index.insert(doc2)

response = query_engine.query("What is Molly?")
print(response)

This is using llama2 served locally using ollama, and fastembed to replace the default openai embeddings llamaIndex would rely on

[zbabarMRMR]

Yes I had almost a similar configurations but still chainlit says you to configure OpenAI key

[willydouhard]

Chainlit does not require an openai key, can you provide the exact error trace?

[zbabarMRMR]

I sorted that out thank you so much