Generator script not working

[killchainrepo]

Trying to run the generator script and it doesn't work. I am using Azure openai/Ai search for index/code space for testing. Please help if possible

• [ x] bug report -> please search issues before submitting
• [ ] feature request
• [x ] documentation issue or request
• [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Any log messages given by the failure

05:06:56 (INFO) scripts: Using Azure OpenAI Service with Azure Developer CLI Credential 05:06:56 (INFO) azure.identity._internal.decorators: AzureDeveloperCliCredential.get_token succeeded 05:06:56 (INFO) scripts: Using Azure Search Service with API Key from AZURE_SEARCH_KEY 05:06:56 (INFO) scripts: Generating 50 questions total, 5 per source, based on search results 05:06:56 (INFO) azure.core.pipeline.policies.http_logging_policy: Request URL: 'https://testservcie3.search.windows.net/indexes('index12')/docs/search.post.search?api-version=REDACTED' Request method: 'POST' Request headers: 'Content-Type': 'application/json' 'Content-Length': '27' 'api-key': 'REDACTED' 'Accept': 'application/json;odata.metadata=none' 'x-ms-client-request-id': '822ae6a6-e5ae-11ee-8800-000d3acb1738' 'User-Agent': 'azsdk-python-search-documents/11.4.0 Python/3.11.8 (Linux-6.2.0-1019-azure-x86_64-with-glibc2.31)' A body is sent with the request 05:06:56 (INFO) azure.core.pipeline.policies.http_logging_policy: Response status: 200 Response headers: 'Transfer-Encoding': 'chunked' 'Content-Type': 'application/json; odata.metadata=none; odata.streaming=true; charset=utf-8' 'Content-Encoding': 'REDACTED' 'Vary': 'REDACTED' 'Server': 'Microsoft-IIS/10.0' 'Strict-Transport-Security': 'REDACTED' 'Preference-Applied': 'REDACTED' 'OData-Version': 'REDACTED' 'request-id': '822ae6a6-e5ae-11ee-8800-000d3acb1738' 'elapsed-time': 'REDACTED' 'Date': 'Tue, 19 Mar 2024 05:06:56 GMT' Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/workspaces/ai-rag-chat-evaluator/scripts/main.py", line 6, in app() File "/home/vscode/.local/lib/python3.11/site-packages/typer/main.py", line 328, in call raise e File "/home/vscode/.local/lib/python3.11/site-packages/typer/main.py", line 311, in call return get_command(self)(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/typer/core.py", line 778, in main return _main( ^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/typer/core.py", line 216, in _main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/vscode/.local/lib/python3.11/site-packages/typer/main.py", line 683, in wrapper return callback(**use_params) # type: ignore ^^^^^^^^^^^^^^^^^^^^^^ File "/workspaces/ai-rag-chat-evaluator/scripts/cli.py", line 40, in generate generate_test_qa_data( File "/workspaces/ai-rag-chat-evaluator/scripts/generate.py", line 37, in generate_test_qa_data logger.info("Processing search document %s", doc["sourcepage"])

Expected/desired behavior

na

OS and Version?

windows 10 VM

Mention any other details that might be useful

na

[pamelafox]

Hm, it seems like the final error code is cut off, but I'm wondering if your search fields are different than its expecting. What fields are in your index?

[killchainrepo]

Hi Pamela

Thanks for looking into this

So the final error code is : KeyError: 'sourcepage' .Sorry for missing this one.

Attaching the Azure ai search field screenshot. Does the generator script requires particular field values?

[pamelafox]

It does assume particular values, but I'll make those configurable. The "sourcepage" field should be replaced with whatever field in your index is used for citations - I'm assuming eiher url or filepath. Replace it here: https://github.com/Azure-Samples/ai-rag-chat-evaluator/blob/main/scripts/generate.py#L37

and here: https://github.com/Azure-Samples/ai-rag-chat-evaluator/blob/f4ab4e1306e4e6f67db137d4bb1f18c32f15b28d/scripts/generate.py#L47

[killchainrepo]

Thanks so much Pamela. It works now.