Benchmarking fails when using Azure OpenAI

aiverify-foundation / moonshot

Moonshot - A simple and modular tool to evaluate and red-team any LLM application.

https://aiverify-foundation.github.io/moonshot/

Apache License 2.0

177 stars 34 forks source link

Benchmarking fails when using Azure OpenAI #328

Open harsh-garg98 opened 2 months ago

harsh-garg98 commented 2 months ago

[Benchmarking] Failed to calculate metrics in executing recipe due to error: The api_key client option must be set either by passing api_key to the client or by setting the TOGETHER_API_KEY environment variable

I am using an Azure OpenAI deployment and passing the API Key in the web UI. Please help me debug the issue.

imda-jacksonboey commented 2 months ago

Hi @harsh-garg98 , you might want to check again if you are using any of the cookbooks as we are only supporting certain cookbook / recipes with specific connectors example:

you may refer to this link for more information on this error : https://aiverify-foundation.github.io/moonshot/faq/

yawen-d commented 2 months ago

Is there any plans to expand the number of connectors of those cookbooks? Especially MLCommons AI Safety Benchmarks?

miyamaya9 commented 2 months ago

hi @yawen-d Currently, the connector used is the one recommended by MLCommons. Expanding the number of connectors will depend on whether MLCommons decides to support additional connectors in the future. We're closely following any updates from their side, and we'll ensure our cookbooks are aligned with their recommendations. Hope this clarifies!