Parsing Error and not showing answer.

[dilzilla]

When I asked a question more often than not, I get the following parsing error. This time I also was not shown the answer to my question and when I asked it again it said it had already answered me and then it have the parsing error and then started to repeat itself in a code block followed by another parsing error.

I am running on a Mac.

These are the errors in the docker terminal. 2024-04-06 22:12:29 chromadb-1 | ERROR: [07-04-2024 03:12:29] Giving up send_request(...) after 4 tries (requests.exceptions.ConnectionError: HTTPSConnectionPool(host='us-api.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xffff6b612750>: Failed to establish a new connection: [Errno 111] Connection refused'))) 2024-04-06 22:12:30 chromadb-1 | INFO: [07-04-2024 03:12:30] Backing off send_request(...) for 1.0s (requests.exceptions.ConnectionError: HTTPSConnectionPool(host='us-api.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xffff6b721250>: Failed to establish a new connection: [Errno 111] Connection refused'))) 2024-04-06 22:12:31 chromadb-1 | INFO: [07-04-2024 03:12:31] Backing off send_request(...) for 0.6s (requests.exceptions.ConnectionError: HTTPSConnectionPool(host='us-api.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xffff6b723e10>: Failed to establish a new connection: [Errno 111] Connection refused'))) 2024-04-06 22:12:32 chromadb-1 | INFO: [07-04-2024 03:12:32] Backing off send_request(...) for 1.8s (requests.exceptions.ConnectionError: HTTPSConnectionPool(host='us-api.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xffff6b51cf50>: Failed to establish a new connection: [Errno 111] Connection refused'))) 2024-04-06 22:12:33 chromadb-1 | ERROR: [07-04-2024 03:12:33] Giving up send_request(...) after 4 tries (requests.exceptions.ConnectionError: HTTPSConnectionPool(host='us-api.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xffff72ab6750>: Failed to establish a new connection: [Errno 111] Connection refused')))

[duronxx]

me too

[nilsherzig]

Parsing errors happen if your LLM isn't responding in the requested format. A format is something like Do i need to use a Tool? No AI: [answer]. Mistral forgot to add the AI: part, which results in a parsing error.

Could you try running the default LLM? Most LLMs which aren't trained to respond with structured output will fail at some point. (btw, the default LLM is a trained version of mistral).

[nilsherzig]

Your @dilzilla error logs are chroma db's attempts to post some analytics data to their API, but your network is blocking this.

[nilsherzig]

@duronxx please open a new issue, this seems unrelated. Are you running LLocalSearch behind a VPN, or in a datacenter?

[dilzilla]

Parsing errors happen if your LLM isn't responding in the requested format. A format is something like Do i need to use a Tool? No AI: [answer]. Mistral forgot to add the AI: part, which results in a parsing error.

Could you try running the default LLM? Most LLMs which aren't trained to respond with structured output will fail at some point. (btw, the default LLM is a trained version of mistral).

Okay what is the correct model to run it with? Also realized I have a custom DNS at the office and that might be blocking some chroma db. I will run it at my house and see if it works better.
@nilsherzig Update: Tried this at home with the hermes-2-pro-mistral-7b model and getting the same issues.

[nilsherzig]

Parsing errors happen if your LLM isn't responding in the requested format. A format is something like Do i need to use a Tool? No AI: [answer]. Mistral forgot to add the AI: part, which results in a parsing error.

Could you try running the default LLM? Most LLMs which aren't trained to respond with structured output will fail at some point. (btw, the default LLM is a trained version of mistral).

Okay what is the correct model to run it with? Also realized I have a custom DNS at the office and that might be blocking some chroma db. I will run it at my house and see if it works better.
@nilsherzig Update: Tried this at home with the hermes-2-pro-mistral-7b model and getting the same issues.

It's perfectly fine to block the chroma DB analytics.

Getting the same parsing error with hermes-2-pro is surprising tho, I haven't gotten a single parsing error with this LLM. Which quant are you using?

I still think I should make the parsing error more understandable to the user.

[dilzilla]

Parsing errors happen if your LLM isn't responding in the requested format. A format is something like Do i need to use a Tool? No AI: [answer]. Mistral forgot to add the AI: part, which results in a parsing error. Could you try running the default LLM? Most LLMs which aren't trained to respond with structured output will fail at some point. (btw, the default LLM is a trained version of mistral).

Okay what is the correct model to run it with? Also realized I have a custom DNS at the office and that might be blocking some chroma db. I will run it at my house and see if it works better. @nilsherzig Update: Tried this at home with the hermes-2-pro-mistral-7b model and getting the same issues.

It's perfectly fine to block the chroma DB analytics.

Getting the same parsing error with hermes-2-pro is surprising tho, I haven't gotten a single parsing error with this LLM. Which quant are you using?

I still think I should make the parsing error more understandable to the user.

@nilsherzig I am using the 8q version. Your readme doesn't specify what one to use.

[nilsherzig]

Parsing errors happen if your LLM isn't responding in the requested format. A format is something like Do i need to use a Tool? No AI: [answer]. Mistral forgot to add the AI: part, which results in a parsing error. Could you try running the default LLM? Most LLMs which aren't trained to respond with structured output will fail at some point. (btw, the default LLM is a trained version of mistral).

Okay what is the correct model to run it with? Also realized I have a custom DNS at the office and that might be blocking some chroma db. I will run it at my house and see if it works better. @nilsherzig Update: Tried this at home with the hermes-2-pro-mistral-7b model and getting the same issues.

It's perfectly fine to block the chroma DB analytics.

Getting the same parsing error with hermes-2-pro is surprising tho, I haven't gotten a single parsing error with this LLM. Which quant are you using?

I still think I should make the parsing error more understandable to the user.

@nilsherzig I am using the 8q version. Your readme doesn't specify what one to use.

The q8 version is the hardcoded default. Did you change anything else? I'm currently building an end to end test pipeline for this project and this LLM manages to solve the "obsidian sync price" question in 94 / 100 runs.

[digitalw00t]

I'm getting the same issue, after doing just a git clone and docker compose up. I get search returns, but they all say the llm is the wrong output and keeps looping looking for more answers.