[Bug]: llama_index.llms.ollama httpx.HTTPStatusError: Server error '500 Internal Privoxy Error' for url 'http://localhost:11434/api/chat'

[Lyy0617]

Bug Description

I am building my local question-answering system using LlamaIndex, where LLM is deployed through Ollama. Initially, it worked fine, but when I started the global Privoxy, there were issues with LLM based on Ollama. However, deploying the local embedding model based on HuggingFaceEmbedding did not encounter any problems. The specific error message is: httpx.HTTPStatusError: Server error '500 Internal Privoxy Error' for url 'http://localhost:11434/api/chat' For more information, check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500

Version

0.10.55

Steps to Reproduce

embed_args = {'model_name': './bce-embedding-base_v1', 'max_length': 512, 'embed_batch_size': 32, 'device': 'cuda'} embed_model = HuggingFaceEmbedding(**embed_args) #token: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 reranker_model = SentenceTransformerRerank( top_n = 5, model = "./bce-reranker-base_v1", device='cuda' ) llm = Ollama(model="qwen2:latest", request_timeout=60.0) Settings.embed_model=embed_model Settings.llm = llm

db = chromadb.PersistentClient(path="./ChromaDB") chroma_collection = db.get_or_create_collection("MyDocuments") vector_store = ChromaVectorStore(chroma_collection=chroma_collection) index = VectorStoreIndex.from_vector_store(vector_store)

vector_retriever = VectorIndexRetriever(index=index, similarity_top_k=5) response_synthesizer = get_response_synthesizer(

llm=llm,

response_mode="tree_summarize",
streaming=True 

) query_engine = RetrieverQueryEngine( retriever=vector_retriever, response_synthesizer=response_synthesizer, node_postprocessors=[reranker_model], ) query_engine.query(“XX”).print_response_stream()

Relevant Logs/Tracbacks

No response

[logan-markewich]

Is that the correct Ollama url? If you are using some proxy, you might need to update the base url

[Lyy0617]

Is that the correct Ollama url? If you are using some proxy, you might need to update the base url

Thank you so much! When I don't use a Proxy, everything works fine, and Ollama is running in the background. So, should I use its absolute path now?

[logan-markewich]

Not sure what you mean by absolute path? I just meant that if you are using a proxy, http://localhost:11434 might not be the correct URL anymore right?

You can modify this like llm = Ollama(..., base_url="http://localhost:11434")

[Lyy0617]

Not sure what you mean by absolute path? I just meant that if you are using a proxy, http://localhost:11434 might not be the correct URL anymore right?

You can modify this like llm = Ollama(..., base_url="http://localhost:11434")

You are right, after enabling the global proxy, http://localhost:11434 is not the correct one. However, I am not sure now where the correct port of Ollama is.