Closed SamuelBoulanger closed 1 year ago
@SamuelBoulanger Did you figure out what was the issue? facing the same issue
Getting the same issue - the index with that name needs to exist on your search service. I tried simply creating the index, but then it gives me other errors like " This index must have valid semantic configurations defined before using the 'semanticConfiguration' query parameter."
Does anyone know where the script is to manually create this index ?
同样的问题
@durichbojan @Thavash I had to add a role to my user in the azure search (search index data contributor) and then redeploy using phsh .\scripts\prepdocs.ps1
Glad to hear it works for you, @SamuelBoulanger. The issue where you have to add roles and re-run often happens if the initial run of azd up errors out for some reason and doesn't complete assigning roles or running the prepdocs script. Closing since this is resolved. @durichbojan @Thavash feel free to reactivate if this is still causing trouble after you do the same thing @SamuelBoulanger described.
Hi, I got it working but just in case anyone else is having the problem - I checked the permissions as per the above and my account was already search index data contributor. So I made my account "owner" ( I know that this is not recommended for production ) on the search index and ran "azd up" again, and now everything is working.
@Thavash Actually I had to redeploy it to one of my colleague's demo environments and the contributor wasn't enough. I followed your suggestion and make my user owner and it works.
I am having the same issue. I have the Search Index Data contributor role assigned. Yet I get error "The index 'gptkbindex' for service 'gptkb-***' was not found" when I try to ask a question to the application. I have tried deleting all resources and re-deploying several times by running 'azd up'. I have also tried assigning the role 'owner' to my account and running 'azd up' but unable to fix the error.
I looked into my storage and did not find any pdfs copied into the content container. So I tried running `python ./scripts/prepdocs.py $cwd/data/ --storageaccount $env:AZURE_STORAGE_ACCOUNT --container $env:AZURE_STORAGE_CONTAINER --searchservice $env:AZURE_SEARCH_SERVICE --index $env:AZURE_SEARCH_INDEX --formrecognizerservice $env:AZURE_FORMRECOGNIZER_SERVICE --tenantid $env:AZURE_TENANT_ID -v manually from the terminal and found out that the imports for all azure libraries and pypdf had to be installed explicitly. After installing those, I am still getting this error.
ImportError: cannot import name 'AzureDeveloperCliCredential' from 'azure.identity'`
Are there any changes or specific imports needed for running from Windows machine?
Any directions you can give are greatly appreciated! Thanks in advance for your help!
"Search Index Data Contributor" role was already there. What fixed it for me was to add "Search Service Contributor" role to my ID. And then run -> phsh .\scripts\prepdocs.ps1
Hope it helps anyone still facing the issue.
Search Service Contributor
Command is pwsh .\scripts\prepdocs.ps1, not phsh
I am having the same issue. Even after I assigned my user the role in Search and Storage service, I am still facing the same thing. I am using a Mac btw, is there any specific thing we need to do after role assignment on mac.? Also when I try to run prepdocs.ps1 , I get a bunch of syntax errors
Hi, I got it working but just in case anyone else is having the problem - I checked the permissions as per the above and my account was already search index data contributor. So I made my account "owner" ( I know that this is not recommended for production ) on the search index and ran "azd up" again, and now everything is working.
This totally worked for me too ( setting owner in extra privilaged assignments), although as said, its not recommended.
I did all of the above now it showing that there is an error with the Certificate. Here is the error log :
Ensuring search index gptkbindex exists
Traceback (most recent call last):
File "E:\Projects\sunrise\git\test\scripts\prepdocs.py", line 299, in <module>
create_search_index()
File "E:\Projects\sunrise\git\test\scripts\prepdocs.py", line 237, in create_search_index
if args.index not in index_client.list_index_names():
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\paging.py", line 124, in __next__
return next(self._page_iterator)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\paging.py", line 76, in __next__
self._response = self._get_next(self.continuation_token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\search\documents\indexes\_generated\operations\_indexes_operations.py", line 512, in get_next
pipeline_response = self._client._pipeline.run( # type: ignore # pylint: disable=protected-access
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 202, in run
return first_node.send(pipeline_request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send
response = self.next.send(request)
^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send
response = self.next.send(request)
^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send
response = self.next.send(request)
^^^^^^^^^^^^^^^^^^^^^^^
[Previous line repeated 2 more times]
File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\policies\_redirect.py", line 156, in send
response = self.next.send(request)
^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\policies\_retry.py", line 470, in send
raise err
File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\policies\_retry.py", line 448, in send
response = self.next.send(request)
^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\policies\_authentication.py", line 113, in send
response = self.next.send(request)
^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send
response = self.next.send(request)
^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send
response = self.next.send(request)
^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send
response = self.next.send(request)
^^^^^^^^^^^^^^^^^^^^^^^
[Previous line repeated 1 more time]
File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 101, in send
self._sender.send(request.http_request, **request.context.options),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\transport\_requests_basic.py", line 364, in send
raise error
azure.core.exceptions.ServiceRequestError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1002)
Hi folks,
I have found the problem: in your environment there is a env variable called $env:AZD_PREPDOCS_RAN. This variable will be set from "false" to "true" after the first run of your deployment. This causes the script which has to build the index to never run again. Set the env variable back to "false" in your terminal:
azd env set AZD_PREPDOCS_RAN "false"
After that check if it's set correctly by this command:
azd env get-values
And after that run the script again (be careful, use this command in your init dev env or absolut path!!!) :
./scripts/prepdocs.sh
This fixed it for me. ;-)
The last script is the one which causes the problem of rebuilding the index.
Same issue with me. Role assignments are correct. I found that the indexes are not getting created, and while azd up, failing on few steps with below errors -
Start-Process: C:\Users\QTAdmin\Documents\WebCode\QutieBot\scripts\prepdocs.ps1:26
Line |
26 | Start-Process -FilePath ($pythonCmd).Source -ArgumentList "-m venv ./ …
| ~~~~~~~
| Cannot validate argument on parameter 'FilePath'. The argument is null or empty. Provide an argument that is not
| null or empty, and then try the command again.
Installing dependencies from "requirements.txt" into virtual environment
Start-Process: C:\Users\QTAdmin\Documents\WebCode\QutieBot\scripts\prepdocs.ps1:35
Line |
35 | Start-Process -FilePath $venvPythonPath -ArgumentList "-m pip install …
| ~~~~~~~~~~~~~~~~~
| This command cannot be run due to the error: The system cannot find the file specified.
Running "prepdocs.py"
Start-Process: C:\Users\QTAdmin\Documents\WebCode\QutieBot\scripts\prepdocs.ps1:39
Line |
39 | Start-Process -FilePath $venvPythonPath -ArgumentList "./scripts/prep …
| ~~~~~~~~~~~~~~~~~
| This command cannot be run due to the error: The system cannot find the file specified.
Thoughts?
I did all of the above now it showing that there is an error with the Certificate. Here is the error log :
Ensuring search index gptkbindex exists Traceback (most recent call last): File "E:\Projects\sunrise\git\test\scripts\prepdocs.py", line 299, in <module> create_search_index() File "E:\Projects\sunrise\git\test\scripts\prepdocs.py", line 237, in create_search_index if args.index not in index_client.list_index_names(): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\paging.py", line 124, in __next__ return next(self._page_iterator) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\paging.py", line 76, in __next__ self._response = self._get_next(self.continuation_token) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\search\documents\indexes\_generated\operations\_indexes_operations.py", line 512, in get_next pipeline_response = self._client._pipeline.run( # type: ignore # pylint: disable=protected-access ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 202, in run return first_node.send(pipeline_request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\policies\_redirect.py", line 156, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\policies\_retry.py", line 470, in send raise err File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\policies\_retry.py", line 448, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\policies\_authentication.py", line 113, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ [Previous line repeated 1 more time] File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 101, in send self._sender.send(request.http_request, **request.context.options), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\transport\_requests_basic.py", line 364, in send raise error azure.core.exceptions.ServiceRequestError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1002)
Did you find any solutions to this? I am facing the same error
For me it's a good way to use GitHub Codespaces to solve all the described problems with certificates, deployment issues, etc. Be sure that you are in the right, initiated directory, before using "azd up" !
I did all of the above now it showing that there is an error with the Certificate. Here is the error log :
Ensuring search index gptkbindex exists Traceback (most recent call last): File "E:\Projects\sunrise\git\test\scripts\prepdocs.py", line 299, in <module> create_search_index() File "E:\Projects\sunrise\git\test\scripts\prepdocs.py", line 237, in create_search_index if args.index not in index_client.list_index_names(): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\paging.py", line 124, in __next__ return next(self._page_iterator) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\paging.py", line 76, in __next__ self._response = self._get_next(self.continuation_token) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\search\documents\indexes\_generated\operations\_indexes_operations.py", line 512, in get_next pipeline_response = self._client._pipeline.run( # type: ignore # pylint: disable=protected-access ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 202, in run return first_node.send(pipeline_request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ [Previous line repeated 2 more times] File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\policies\_redirect.py", line 156, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\policies\_retry.py", line 470, in send raise err File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\policies\_retry.py", line 448, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\policies\_authentication.py", line 113, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 70, in send response = self.next.send(request) ^^^^^^^^^^^^^^^^^^^^^^^ [Previous line repeated 1 more time] File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\_base.py", line 101, in send self._sender.send(request.http_request, **request.context.options), ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\Projects\sunrise\git\test\scripts\.venv\Lib\site-packages\azure\core\pipeline\transport\_requests_basic.py", line 364, in send raise error azure.core.exceptions.ServiceRequestError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1002)
Did you find any solutions to this? I am facing the same error
@Lohrran still not @CBeermann I tried your solution it didn't work unfortunately
Please use the GitHub Codespaces to solves this problem. I had the same problem at first, but from the Codespaces it worked fine. Be sure that you are in the initiated directory and not in the project root dir. ;-)
what worked for me is setting up the environment variable REQUESTS_CA_BUNDLE to the path of CA bundle certificate using this command azd env set REQUESTS_CA_BUNDLE "path" the path is diffrent for every OS you can find how and where here here The main point was that I was using a proxy and needed to add my certificate in the pem file at the end. I hope this will help you.
I am experiencing the same issues:
Error: () The index 'gptkbindex' for service 'gptkb-ggsebkchwjd7c' was not found. Code: Message: The index 'gptkbindex' for service 'gptkb-ggsebkchwjd7c' was not found.
Additionally, when executing the 'prepdocs.py' command (pwsh .\scripts\prepdocs.ps1)in the command prompt, I am encountering the following error: prepdocs.py: error: unrecognized arguments.
I have already checked my roles and have assigned myself as the owner, but I am still encountering the first error.
Any help on this matter would be greatly appreciated.
@Ukhan11 Can you confirm the index exists when you visit the search service in the Portal? eg.
Then can you check Access control for the search service to make sure you have all the expected roles?
For the unrecognized arguments error, is there any more detail than that?
@pamelafox
there is nothing in my index page I have followed this link https://msandbu.org/how-to-setup-azure-openai-with-chatgpt-using-your-own-data/
Can you tell me what steps is missing in my deployments?
the roles I have:
@Ukhan11 Can you confirm the index exists when you visit the search service in the Portal? eg.
Then can you check Access control for the search service to make sure you have all the expected roles?
For the unrecognized arguments error, is there any more detail than that?
Hi, have you had a chance to look into the error issues?
If the index does not exist, then the postprovision hook did not work. It should run prepdocs.py which should create the index. Please try running either prepdocs.sh
or prepdocs.ps1
and see if there are any errors.
If the index does not exist, then the postprovision hook did not work. It should run prepdocs.py which should create the index. Please try running either
prepdocs.sh
orprepdocs.ps1
and see if there are any errors.
I have run the command : pwsh .\scripts\prepdocs.ps1 but getting the following error:
er==3.2.1->-r ./scripts/requirements.txt (line 4)) (3.2.2) Running "prepdocs.py" usage: prepdocs.py [-h] [--category CATEGORY] [--skipblobs] [--storageaccount STORAGEACCOUNT] [--container CONTAINER] [--storagekey STORAGEKEY] [--tenantid TENANTID] [--searchservice SEARCHSERVICE] [--index INDEX] [--searchkey SEARCHKEY] [--remove] [--removeall] [--localpdfparser] [--formrecognizerservice FORMRECOGNIZERSERVICE] [--formrecognizerkey FORMRECOGNIZERKEY] [--verbose] files prepdocs.py: error: unrecognized arguments: Open ai/data/*
@Ukhan11 That issue is discussed here: https://github.com/Azure-Samples/azure-search-openai-demo/issues/244 The code currently doesn't work with folder names that have spaces in it. If you can, please rename your folder.
@Ukhan11 That issue is discussed here: #244 The code currently doesn't work with folder names that have spaces in it. If you can, please rename your folder.
Thank you for your assistance, that solution works for me.
@Ukhan11 That issue is discussed here: #244 The code currently doesn't work with folder names that have spaces in it. If you can, please rename your folder.
Additionally, could you kindly guide me on the process of uploading our own documents instead of utilizing this one? I am able to create index dynamically but struggling with how we can upload our won documents in the index.
Thanks in advance
@Ukhan11 See the FAQ re uploading more data: https://github.com/Azure-Samples/azure-search-openai-demo#faq
I have the same issue (no indexes in search service created) and when I run pwsh .\scripts\prepdocs.ps1
I get following error: prepdocs.py: error: argument --openaikey: expected one argument
What can be the problem?
Thank you
@Marty0427 Thanks for the report, I've put a fix in https://github.com/Azure-Samples/azure-search-openai-demo/pull/651
I'm done with the deployment but when using the app (chat or ask question) I got this error message. Looking into the ressource there's no index that have been created.
Error: () The index 'gptkbindex' for service 'gptkb-d3g3thoxsz5ve' was not found. Code: Message: The index 'gptkbindex' for service 'gptkb-d3g3thoxsz5ve' was not found.