Synthesizer - generate from scratch returns only one golden

confident-ai / deepeval

The LLM Evaluation Framework

https://docs.confident-ai.com/

Apache License 2.0

3.75k stars 298 forks source link

Synthesizer - generate from scratch returns only one golden #972

Open attilalaszlo opened 3 months ago

attilalaszlo commented 3 months ago

Hey, I think I found a bug I would like to report. Because of an indentation typo, synchronous version of the generate_goldens_from_scratch method will use only one initial golden, and then return with the evolved versions of only that. https://github.com/confident-ai/deepeval/blob/095e37e5372be82d68be846e74b5deaa804ef116/deepeval/synthesizer/synthesizer.py#L562

Works for async version.

attilalaszlo commented 3 months ago

somewhat related: if num_evolutions=0 then the progress message is: ✨ 🍰 ✨ You're generating up to 0 goldens using DeepEval's latest Synthesizer Which is not true, it will generate num_initial_goldens goldens only won't be evolve it with evolutions.

kritinv commented 2 months ago

Should be fixed in the next synthesizer update soon.