search

confident-ai / deepeval

The LLM Evaluation Framework
https://docs.confident-ai.com/
Apache License 2.0
3.83k stars 308 forks source link

Do you support evaluation on multi-lingual model and dataset? #718

Closed petergaoshan closed 6 months ago

petergaoshan commented 7 months ago

Can I evaluate other languages other than English using this package?

penguine-ip commented 7 months ago

@petergaoshan Depends if the evaluation model you're using supports that language. You an use any evaluation you wish using deepeval

EssamWisam commented 4 months ago

@penguine-ip I am using a model that supports my language yet some questions generated are in English despite all text being in the other language. The solution seems to be to modify the internal prompt but this is not exposed.

penguine-ip commented 4 months ago

@EssamWisam happy to accept a PR on this, i'm imagining an enum of languages and provide it during metrics initialization before passing it onto the prompt template.

EssamWisam commented 4 months ago

@EssamWisam happy to accept a PR on this, i'm imagining an enum of languages and provide it during metrics initialization before passing it onto the prompt template.

OK I will try to find time for this. My complain was actually regarding test set generation. I'm not assuming metrics will be that language sensitive.