how to use ollama

[jcl2023]

My system has ollama server running. But I got the following error when I configure tldw to use ollama API

ollama_api_IP = http://127.0.0.1:11434/api/generate ollama_api_key = ollama_model = qwen2.5:7b-instruct-q5_K_M

WARNING:root:Summarization with ollama failed: Error: Invalid API Name ollama WARNING:root:Failed to generate summary using ollama API INFO:root:Summarization completed successfully. ERROR:root:Summarization failed.

Any idea why this happens?

Thanks,

[rmusser01]

Thank you for opening an issue, sorry about that. Seems I forgot to update the summarize function with ollama + custom OpenAI API endpoints. Should be fixed now, please update and try again.

[jcl2023]

I got another error after pulling your fix:

INFO:root:process_url_with_metadata: Starting summarization with ollama... INFO:root:Starting summarization process... INFO:root:Ollama: Using API key from config file Ollama: Submitting request to API endpoint ERROR:root:Ollama: Error in processing: 'content'

Thanks,

[rmusser01]

Thanks for the update, sorry that its still not working, I refuse to use ollama, so I am doing this blind. I've gone ahead and modified the JSON key check to look for '['message']['content']` in the response, instead of just 'content'. That should resolve the issue, according to their docs. https://github.com/ollama/ollama/blob/main/docs/api.md

If it doesn't please let me know and I'll continue to try and figure it out.

[jcl2023]

It still doesn't work.

To verify my ollama server is OK, I use curl (below) and it works OK:

curl http://192.168.0.224:11434/api/generate -d '{ "model": "qwen2.5-coder:7b-instruct-q5_K_M", "prompt": "Why is the sky blue?", "stream": false }'

Do you verify tldw with lamma.cpp server? If yes, I can try llama.cpp server.

Thanks,

[rmusser01]

I did not test it recently, just did, found bugs, fixed them, and pushed the changes. Llama.cpp should work for chatting now with no issues.

Did you receive the same error as before? Can you please post the first bit of the servers response to that curl command?

[jcl2023]

Here is the error from printouts: Ollama: Submitting request to API endpoint ERROR:root:Ollama: Error in processing: 'content' INFO:root:Summarization with ollama successful INFO:root:Summary generated using ollama API INFO:root:Summarization completed successfully. INFO:root:process_url_with_metadata: Saving transcription and summary... INFO:root:process_url_with_metadata: Transcription saved to: Results/Audio_Processing/The History of Toilets_transcription.txt INFO:root:process_url_with_metadata: Summary saved to: Results/Audio_Processing/The History of Toilets_summary.txt INFO:root:process_url_with_metadata: Keywords prepared: ['default', 'no_keyword_set'] INFO:root:process_url_with_metadata: Content updated successfully for media ID: 6. New version: 13 INFO:root:Attempting to read file: Results/Audio_Processing/The History of Toilets_summary.txt INFO:root:Detected encoding: ascii INFO:root:Successfully decoded file with encoding: ascii

The response of my curl command:

{"model":"qwen2.5-coder:7b-instruct-q5_K_M","created_at":"2024-10-13T01:49:45.978756425Z","response":"The sky appears blue because of a phenomenon called Rayleigh scattering, named after Lord Rayleigh who first described it in 1871.\n\nWhen sunlight enters Earth's atmosphere, it encounters gas molecules such as nitrogen and oxygen. These molecules are much smaller than the wavelength of visible light, so they scatter the light in all directions.\n\nThe amount of scattering depends on the wavelength of the light. Shorter wavelengths (such as blue) are scattered more easily than longer wavelengths (such as red). This is because shorter wavelengths interact more strongly with gas molecules and are therefore more likely to be scattered.\n\nAs a result, when sunlight enters Earth's atmosphere, it is scattered in all directions, but the shorter wavelengths of blue light are scattered more frequently. As a result, we see the sky as blue during the day.\n\nDuring sunrise and sunset, the sun is lower on the horizon, so its light has to travel through more of the Earth's atmosphere before reaching our eyes. This causes even more scattering of the shorter wavelength blue light, resulting in a reddish or orange hue in the sky during these times.","done":true,"done_reason":"stop","context":[151644,8948,198,2610,525,264,10950,17847,13,151645,198,151644,872,198,10234,374,279,12884,6303,30,151645,198,151644,77091,198,785,12884,7952,6303,1576,315,264,24844,2598,13255,62969,71816,11,6941,1283,10214,13255,62969,879,1156,7481,432,304,220,16,23,22,16,382,4498,39020,28833,9237,594,16566,11,432,33906,6819,34615,1741,438,46403,323,23552,13,4220,34615,525,1753,9155,1091,279,45306,315,9434,3100,11,773,807,44477,279,3100,304,678,17961,382,785,3311,315,71816,13798,389,279,45306,315,279,3100,13,10698,261,92859,320,20805,438,6303,8,525,36967,803,6707,1091,5021,92859,320,20805,438,2518,568,1096,374,1576,23327,92859,16282,803,16510,448,6819,34615,323,525,8916,803,4363,311,387,36967,382,2121,264,1102,11,979,39020,28833,9237,594,16566,11,432,374,36967,304,678,17961,11,714,279,23327,92859,315,6303,3100,525,36967,803,13814,13,1634,264,1102,11,582,1490,279,12884,438,6303,2337,279,1899,382,16014,63819,323,42984,11,279,7015,374,4722,389,279,34074,11,773,1181,3100,702,311,5821,1526,803,315,279,9237,594,16566,1573,18739,1039,6414,13,1096,11137,1496,803,71816,315,279,23327,45306,6303,3100,11,12942,304,264,62144,812,476,18575,39040,304,279,12884,2337,1493,3039,13],"total_duration":77062695133,"load_duration":73682358729,"prompt_eval_count":25,"prompt_eval_duration":34378000,"eval_count":222,"eval_duration":3343166000}

[rmusser01]

Thank you, I broke down and installed ollama/set it up and verified it works with the chat and summary endpoints.

Now Ollama works. As of Commit: #353 (Uses the OpenAI API endpoint)

Thanks again for the heads up about this.

[jcl2023]

I am not sure this new issue is related to the original one or not. When I use ollama for rag search, I got this:

ERROR:root:Error in enhanced_rag_pipeline: Required inputs (['token_type_ids']) are missing from input feed (['input_ids', 'attention_mask']).

[jcl2023]

It seems it has nothing to do with ollama. I narrowed it down whether exception occurred below: def enhanced_rag_pipeline(query: str, api_choice: str, keywords: str = None) -> Dict[str, Any]:

Rerank the results

        reranked_results = ranker.rerank(rerank_request)

Hope it helps you to find the rootcause

Thanks

[rmusser01]

Can you please file a new issue with the full output of the console while running the application with:

python summarize.py -gui -log DEBUG

I'm unable to replicate the error you posted, as those error messages aren't in my code, and I don't see that pop up when testing either RAG pipeline with Ollama.