search

Azure-Samples / ai-rag-chat-evaluator

Tools for evaluation of RAG Chat Apps using Azure AI Evaluate SDK and OpenAI
MIT License
222 stars 78 forks source link

HTTP error 431, Request Header Fields Too Large, when running generate.py #99

Closed DuboisABB closed 2 months ago

DuboisABB commented 2 months ago

This issue is for a: (mark with an x)

- [X] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Follow the setup instructions azd auth login python -m scripts generate --output=example_input/qa.jsonl --numquestions=10 --persource=5

Any log messages given by the failure

(evalenv) PS C:\Programming\ai-rag-chat-evaluator> python -m scripts generate --output=example_input/qa.jsonl --numquestions=10 --persource=5
[19:55:52] INFO     Using Azure OpenAI Service with Azure Developer CLI Credential           service_setup.py:53
[19:55:53] INFO     Using Azure Search Service with Azure Developer CLI Credential           service_setup.py:81
           INFO     Generating 10 questions total, 5 per source, based on search results          generate.py:23
[19:55:59] INFO     Processing search document 0002-07-0-00010-XXM Detector Modules REV_E.doc     generate.py:36
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Programming\ai-rag-chat-evaluator\scripts\__main__.py", line 6, in <module>
    app()
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\typer\main.py", line 338, in __call__
    raise e
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\typer\main.py", line 321, in __call__
    return get_command(self)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\typer\core.py", line 728, in main
    return _main(
           ^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\typer\core.py", line 197, in _main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\typer\main.py", line 703, in wrapper
    return callback(**use_params)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\scripts\cli.py", line 57, in generate
    generate_test_qa_data(
  File "C:\Programming\ai-rag-chat-evaluator\scripts\generate.py", line 39, in generate_test_qa_data
    result = qa_generator.generate(
             ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\azure\core\tracing\decorator.py", line 94, in wrapper_use_tracer
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\azure\ai\ml\_telemetry\activity.py", line 295, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\azure\ai\generative\synthetic\qa.py", line 397, in generate
    content, token_usage = _completion_with_retries(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\azure\ai\generative\synthetic\qa.py", line 69, in _completion_with_retries
    response = client.chat.completions.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\openai\_utils\_utils.py", line 274, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\openai\resources\chat\completions.py", line 668, in create
    return self._post(
           ^^^^^^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\openai\_base_client.py", line 1260, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\openai\_base_client.py", line 937, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\openai\_base_client.py", line 1041, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.APIStatusError: Error code: 431 - {'error': {'message': 'HTTP code 431 from API (<html>\n  <head>\n    <title>Request Header Fields Too Large</title>\n  </head>\n  <body>\n    <h1><p>Request Header Fields Too Large</p></h1>\n    Error parsing headers: &#x27;limit request headers fields size&#x27;\n  </body>\n</html>\n)', 'type': None, 'param': None, 'code': None}}

Expected/desired behavior

No error.

pamelafox commented 2 months ago

The last time that I saw this error pop up, it was in the azure-search-openai-demo: https://github.com/Azure-Samples/azure-search-openai-demo/issues/1215

The cause there was a "key" that was actually an Entra token that was incredibly long for some reason.

In this repo, this is the relevant code, in service_setup.py, that generates the token:

  api_key = azure_credential.get_token("https://cognitiveservices.azure.com/.default").token
openai_config = {
            "api_type": "azure",
            "api_base": f"https://{os.environ['AZURE_OPENAI_SERVICE']}.openai.azure.com",
            "api_key": api_key,
            "api_version": "2024-02-15-preview",
            "deployment": os.environ["AZURE_OPENAI_EVAL_DEPLOYMENT"],
            "model": os.environ["OPENAI_GPT_MODEL"],
}

You could print the length of api_key value there and see if it is very long. Could you report back what length you see?

DuboisABB commented 2 months ago

Thanks for the pointer. I added a log of the length, looks like it's 9719, which indeed looks excessively long.

(evalenv) PS C:\Programming\ai-rag-chat-evaluator> python -m scripts generate --output=example_input/qa.jsonl --numquestions=10 --persource=5
[08:46:28] INFO     Using Azure OpenAI Service with Azure Developer CLI Credential                   service_setup.py:54
[08:46:29] INFO     API Key Len: 9719                                                                service_setup.py:57
           INFO     Using Azure Search Service with Azure Developer CLI Credential                   service_setup.py:84
           INFO     Generating 10 questions total, 5 per source, based on search results                  generate.py:23

However, thanks to your hint, I added AZURE_OPENAI_KEY and AZURE_SEARCH_KEY to my .env file. Look like I somehow need to enable key based access however, I'll look into it.

  File "C:\Programming\ai-rag-chat-evaluator\evalenv\Lib\site-packages\openai\_base_client.py", line 1041, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.PermissionDeniedError: Error code: 403 - {'error': {'code': 'AuthenticationTypeDisabled', 'message': 'Key based authentication is disabled for this resource.'}}
pamelafox commented 2 months ago

Yes, we disable key-based access by default in the azure-search-openai-demo infra, here: https://github.com/Azure-Samples/azure-search-openai-demo/blob/7b55ad2b4cb4911141a69963a230ee79fd2f2b41/infra/main.bicep#L403 You can set to false and re-run "azd provision".

But I'll investigate why your token might be so long and ask the Generative SDK to use the token provider route.

pamelafox commented 2 months ago

I've filed this issue for QADataGenerator: https://github.com/Azure/azure-sdk-for-python/issues/37050

DuboisABB commented 2 months ago

Thanks again. I set all 3 instances of disableLocalAuth to false: openAi, documentIntelligence (probably not needed) and searchService. I also upgraded azure-search-openai-demo to the latest version and set AZURE_DISABLE_APP_SERVICES_AUTHENTICATION to false. No more openAI error but then I got an unauthorized error for search.

I remembered that I had went into the portal to the search instance to change API Access Control from RBAC only to Both (API keys + RBAC) but running azd up had reset that. I changed it again to both and it finally works.

I then asked copilot for the required change to search-services.bicep to make that change permanent and it came up with this change:

param authOptions object = {
    aadOrApiKey: {
      aadAuthFailureMode: 'http401WithBearerChallenge' // or 'http403'
    }
}

It seem to work. After azd up the API access control setting is still set to Both.

Thanks Pamela for your support. Next time will be to make evaluate work but the authentication is more tricky. I'll update in #97