Open adkakne opened 1 month ago
This issue is caused by expecting JSON output from the LLM. The prompt template provides instructions for the LLM to provide JSON output. However, you cannot expect the model to return JSON every time. Even more complex models like GPT4 can sometimes make mistakes. Contributors could try to solve it by integrating, for example, Instructor https://github.com/jxnl/instructor/ in DeepEval
@penguine-ip Is it possible to take a look at this issue please ?
Describe the bug I encounter 'ValueError: Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.' while using most popular open source chat models in DeepEval framework.
To Reproduce
Expected behavior After running
deepeval test run test_chatbot.py
, I getValueError: Evaluation LLM outputted an invalid JSON. Please use a better evaluation model.
for all 5 opensource models I tested.Desktop (please complete the following information):
Additional context DeepEval version is 0.21.64.