search

Azure-Samples / ai-rag-chat-evaluator

Tools for evaluation of RAG Chat Apps using Azure AI Evaluate SDK and OpenAI
MIT License
209 stars 75 forks source link

mismatched_organization error #76

Open yaqinamy opened 5 months ago

yaqinamy commented 5 months ago

when comparing local llama2 model with GPT4, got error:

ERROR Failed to send a test chat completion to the GPT deployment due to error: evaluate.py:110 Error code: 401 - {'error': {'message': 'OpenAI-Organization header should match organization for API key', 'type': 'invalid_request_error',
'param': None, 'code': 'mismatched_organization'}

pamelafox commented 5 months ago

So are you using llama2 via ollama for the actual evaluation?

yaqinamy commented 5 months ago

Yes Pamela, I am learning from your video using llama2 via ollama which I deployed local and GPT4 deployed at azure.

pamelafox commented 5 months ago

Okay, so to clarify:

The error in your report shows an issue with the evaluation model, which would be GPT-4. But you shouldn't need to specify OpenAI organization for an Azure model. Do the logs indicate whether it's trying to use Azure or OpenAI.com?

You should see the logs from this code:

https://github.com/Azure-Samples/ai-rag-chat-evaluator/blob/7b5d503dc96a5185446b16bd8db24558614d2ec9/scripts/service_setup.py#L14