Validate LLM output

[adeprez]

Object structures extracted from LLM outputs cannot be 100% trusted. Therefore, they must be validated before being handed over to engines.

• [ ] Define a validator for LLM output
• [ ] Raise an error when extraction doesn't match the expected shape

[Apetun]

Hey! I would like to help with this. Can you help me with how to get started?

[dhuynh95]

Thanks @Apetun for the help!

It might be overkill and different/complementary to what @adeprez proposed, but one way would be to add a small open-LLM, like Phi-3, with outlines to ensure a specific format. It is heavy though to use, so I don't know if we want to go that route, especially given that some models like GPT-4o are (relatively) consistent

[dhuynh95]

Maybe what I just mentioned should in another issue, and @adeprez you present what you had in mind? ^^

[adeprez]

Thank you, Apetun!

This task focuses on ensuring that the action returned by the extractor has the correct format, rather than addressing the model output itself.

In the navigation engine, we extract the JSON code block from the LLM output to determine the appropriate action. We currently take this content as-is, but occasionally the output doesn't conform to the expected format. To enhance the safety of its processing in the code, we want to raise an error immediately upon detecting an invalid output from the model.

As a first step, we want to make the navigation action processing safer (lavague.core.navigation.NavigationEngine).

action_shape_validator = [...]  # define how the extracted value should look like
action = self.extractor.extract(response, action_shape_validator)   # error raised if action has the wrong shape

We expect a list of actions:

[
  {
    "action": {
      "name": "str"
    }
    "args": { }
  }
]

Later, extractors will return an object instead of a raw str. But it will have impact on other parts of the code so it's a bit more complex.

Note: We will be transitioning to YAML soon (#423), but the concept of object validation will remain relevant.

[Apetun]

Thank you for the clarification. I will look into it.

[adeprez]

Useful resources for output validation using LangChain + Pydantic:

• https://python.langchain.com/v0.1/docs/modules/model_io/output_parsers/types/json/
• https://python.langchain.com/v0.1/docs/modules/model_io/output_parsers/types/yaml/

[dhuynh95]

@adeprez I found this too: https://docs.llamaindex.ai/en/stable/module_guides/querying/structured_outputs/