Fix to allow repeat evals on azureai - Githubissues

UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations

https://inspect.ai-safety-institute.org.uk/

MIT License

627 stars 118 forks source link

Fix to allow repeat evals on azureai #845

Closed ole-jorgensen closed 1 week ago

ole-jorgensen commented 1 week ago

This PR contains:

[ ] New features
[ ] Changes to dev-tools e.g. CI config / github tooling
[ ] Docs
[x] Bug fixes
[ ] Code refactor

What is the current behavior? (You can also link to an open issue here)

Repeated calls to azureai models leads to an async error, which stems from the AzureAI ChatCompletionsClient needing to be manually closed

What is the new behavior?

Creates the client in the generate block, and closes it at the end of the generation. Also added a regression test.

Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

Other information: