Running Ollama with Pydantic models

[ernOho]

Currently not working:

Error Message when running local Ollama lama2 model:

langchain_core.exceptions.OutputParserException: Failed to parse SummaryEvaluationItem from completion {"properties": {"metric": {"title": "Metric", "description": "The name of the metric used", "type": "string"}, "score": {"title": "Score", "description": "The score given based on the metric", "type": "integer"}, "reason": {"title": "Reason", "description": "The provided reason for the given score", "type": "string"}}, "required": ["metric", "score"]}. Got: 2 validation errors for SummaryEvaluationItem
metric
  field required (type=value_error.missing)
score
  field required (type=value_error.missing)

[ernOho]

https://www.reddit.com/r/LangChain/comments/1842ms8/langchain_system_prompting_with_ollama_need_help/

Seems to be a general issue.

Workaround?: https://k33g.hashnode.dev/prompts-and-chains-with-ollama-and-langchain

[Bruno-val-bus]

https://www.reddit.com/r/LangChain/comments/1842ms8/langchain_system_prompting_with_ollama_need_help/

Seems to be a general issue.

Workaround?: https://k33g.hashnode.dev/prompts-and-chains-with-ollama-and-langchain

We can try it with the linked workaround. There is already a similar implementation in services/error_finders.py in the SchemaChainWrapper class. Instead of returning a pydantic model, it return a dict. So this would mean we have to handle it differently in the evaluator class. Can you first try to set the fields in SummaryEvaluationItem to Optional and see what happens? @ernOho

[ernOho]

After setting fields to Optional, the chain.invoke(...) call in SummaryEvaluator, all fields are returned as None (which was to be expected). Thoughts @Bruno-val-bus ?

Ill look into the workaround

[ernOho]

@Bruno-val-bus, problem solved, per you suggested. After checking into the docs, using StructuredOutputParser makes sense for less powerfull models (e.g. llama3:8b) while pydandic parser makes sense for more powerfull ones.

Should investigate using StructuredOutputParser vs Pydantic parser for llama3:70b different sizes.

Or would it make sense to stick to the generic StructuredOutputParser so we don't have to duplicate outputs, as our required outputs are not very complex? ```

[ernOho]

Linked it to the relevant feature branch.