Current build (after pro search) is broken

[arsaboo]

I am seeing with the latest build:

Errors in browser console:

[rashadphz]

Shoot, thanks for catching this, I’ll fix it

[rashadphz]

Should be fixed, feel free to re-open if you run into any more issues.

[arsaboo]

ok...now I am seeing

[arsaboo]

Here's the full stack trace:

Traceback (most recent call last):
  File "/workspace/.venv/lib/python3.11/site-packages/instructor/retry.py", line 158, in retry_sync
    return process_response(
           ^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/instructor/process_response.py", line 142, in process_response
    model = response_model.from_response(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/instructor/function_calls.py", line 115, in from_response
    return cls.parse_tools(completion, validation_context, strict)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/instructor/function_calls.py", line 207, in parse_tools
    return cls.model_validate_json(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/pydantic/main.py", line 580, in model_validate_json
    return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for QueryPlan
steps
  Field required [type=missing, input_value={'query': 'what is farfalle?'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.7/v/missing
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/workspace/src/backend/agent_search.py", line 311, in stream_pro_search_qa
    async for event in stream_pro_search_objects(request, llm, query, session):
  File "/workspace/src/backend/agent_search.py", line 115, in stream_pro_search_objects
    query_plan = llm.structured_complete(
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/src/backend/llm/base.py", line 53, in structured_complete
    return self.client.chat.completions.create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/instructor/client.py", line 91, in create
    return self.create_fn(
           ^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/instructor/patch.py", line 143, in new_create_sync
    response = retry_sync(
               ^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/instructor/retry.py", line 152, in retry_sync
    for attempt in max_retries:
  File "/workspace/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 435, in __iter__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 368, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 410, in exc_check
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/workspace/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 183, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/workspace/.venv/lib/python3.11/site-packages/instructor/retry.py", line 173, in retry_sync
    raise InstructorRetryException(
instructor.retry.InstructorRetryException: 1 validation error for QueryPlan
steps
  Field required [type=missing, input_value={'query': 'what is farfalle?'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.7/v/missing
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/workspace/src/backend/main.py", line 111, in generator
    async for obj in stream_fn(request=chat_request, session=session):
  File "/workspace/src/backend/agent_search.py", line 317, in stream_pro_search_qa
    raise HTTPException(status_code=500, detail=detail)
fastapi.exceptions.HTTPException: 500: 1 validation error for QueryPlan
steps
  Field required [type=missing, input_value={'query': 'what is farfalle?'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.7/v/missing

[rashadphz]

What model are you using? The smaller models can’t do pro search consistently unfortunately.

[arsaboo]

llama3

[arsaboo]

Is there any specific model you want me to try?

[arsaboo]

Ok..it worked with Mistral.

BTW....how do we add any other model? I see we can use LiteLLM, but I couldn't see any documentation. I do have a Litellm setup.

[rashadphz]

Any of the cloud models should work but results for GPT-4o are the best.

In terms local models I was able to get structured output with phi3 but it probably won’t be consistent. I’ll have to look into this more though

[arsaboo]

Phi3, Mistral, and Gemma worked for me.

[rashadphz]

Ok..it worked with Mistral.

BTW....how do we add any other model? I see we can use LiteLLM, but I couldn't see any documentation. I do have a Litellm setup.

I’ll add documentation for this but you can set CUSTOM_MODEL in your .env as “/”. The litellm docs have a list of supported models and providers. You will also need to add any env variable required by the provider.

[arsaboo]

We need a better way to produce the output. This is the output with various models:

Gemma:

Phi3:

Mistral:

All of these are much smaller and less useful than non-expert search. Here's the output with Gemma (non-expert)

[rashadphz]

Hmm, interesting. I suspect it has something to do with the smaller models not being able to handle amount of context used for the Pro search. Thanks for catching this I’ll have to look into it.

[arsaboo]

I would love to try this with some of the larger models. Meanwhile, let us change gemma to gemma2

[rashadphz]

Can you try setting CUSTOM_MODEL=ollama_chat/gemma2 in your .env? Let me know if that works

[rashadphz]

or you can use ollama_chat/gemma2:27b

[rashadphz]

Although I'm seeing it's facing issues right now with ollama: https://github.com/ollama/ollama/issues/5341