[FEATURE] Improve reliability of structured outputs in AI Services

langchain4j / langchain4j

Java version of LangChain

https://docs.langchain4j.dev

Apache License 2.0

4.92k stars 979 forks source link

Open langchain4j opened 8 months ago

langchain4j commented 8 months ago

[x] If LLM responds with a JSON wrapped in backticks and/or some additional text, try to extract that instead of failing
[ ] Automatically enable JSON mode if return type of AI Service is POJO and underlying LLM supports it
[ ] (Configurable) Retry in case of parsing failure, feeding the error back to the LLM
[ ] If return type is enum/boolean, automatically use logitBias if supported by the LLM
[ ] Consider adding function calling as a second option for structured outputs
[ ] Consider using existing libraries to generate/validate JSON schema

ruizrube commented 7 months ago

Hi,

The templates for parsing outputs were developed and tested using ChatGPT, isn't?. Open LLMs, such as Llama, are very more verbose in their answers.

kind regards. Thank you for such a stunning project.

langchain4j commented 7 months ago

@ruizrube yes, most of the high-level features were developed with at least gpt-3.5-turbo-level LLM in mind.

But now when we have llama3/mixtral/etc and json mode in Ollama, it should work as good for most of the use cases.

But we need to implement this PR anyway to ensure that a long tail of corner cases is handled well.

langchain4j commented 6 months ago

abbas-gadhia commented 6 months ago

In case anyone is stuck with this, just enable the json mode of the LLM for the time being.