search

eyurtsev / kor

LLM(😽)
https://eyurtsev.github.io/kor/
MIT License
1.6k stars 88 forks source link

when there are no examples provided, the parsing results are inaccurate #202

Closed aixiamomo closed 1 year ago

aixiamomo commented 1 year ago 
res = chain.run("Please help me search for all the tender notices of big data companies on record until the day before yesterday.")

{'data': {}, 'raw': '{\n "industry": "big data",\n "type": "tender",\n "deadline": "2023-08-05"\n}', 'errors': [ParseError('The LLM has returned structured data which does not match the expected schema. Providing additional examples may help improve the parse.')], 'validated_data': {}}

the top-level object is missing

eyurtsev commented 1 year ago

Hi @aixiamomo -- this is not a bug in kor, but a limitation of the LLM that you're using. Most LLMs can't parse structured data well without having some examples.

See guidelines here: https://eyurtsev.github.io/kor/guidelines.html

Closing as this is not a bug that needs to be fixed.

aixiamomo commented 1 year ago

Hi @aixiamomo -- this is not a bug in kor, but a limitation of the LLM that you're using. Most LLMs can't parse structured data well without having some examples.

See guidelines here: https://eyurtsev.github.io/kor/guidelines.html

Closing as this is not a bug that needs to be fixed.

Thank you, I understand. Meanwhile, I have used the following code to ensure that the llms can produce output in the desired format even without examples:

from kor import prompts, examples

def _generate_examples(node):
    examples_list = examples.generate_examples(node)
    return examples_list if examples_list else [("", {node.id: {}})]

prompts.generate_examples = _generate_examples