Return `None` if the function calling step returns an invalid assessment

citadel-ai / langcheck

Simple, Pythonic building blocks to evaluate LLM applications.

https://langcheck.readthedocs.io/en/latest/index.html

MIT License

186 stars 17 forks source link

Return `None` if the function calling step returns an invalid assessment #124

Closed yosukehigashi closed 5 months ago

yosukehigashi commented 5 months ago

Updates the eval clients to set a metric value to None in the rare case that the function calling step returns an invalid assessment. Otherwise, the evaluator will fail with a KeyError.

(I tried for a bit to trigger a bad assessment but couldn't get it to happen 😅)

Also bump the version to 0.8.0.dev3