Can't use self signed certificate (using Docker container for Cosmos DB) with 001-quickstart samples

Rom1deTroyes commented 1 year ago

Hello,

I try to run the samples here with a Docker image of Cosmos DB, taken from : https://learn.microsoft.com/en-us/azure/cosmos-db/docker-emulator-linux?tabs=sql-api%2Cssl-netstd21

Following the README, I get this error :

python 001-quickstart/app.py
Traceback (most recent call last):
  File "/home/romain/Projects/cosmos-db-nosql-python-samples/001-quickstart/app.py", line 27, in <module>
    database = client.create_database_if_not_exists(id=DATABASE_NAME)
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/cosmos/cosmos_client.py", line 319, in create_database_if_not_exists
    database_proxy.read(
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/cosmos/database.py", line 141, in read
    self._properties = self.client_connection.ReadDatabase(
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/cosmos/_cosmos_client_connection.py", line 342, in ReadDatabase
    return self.Read(path, "dbs", database_id, None, options, **kwargs)
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/cosmos/_cosmos_client_connection.py", line 2201, in Read
    result, self.last_response_headers = self.__Get(path, request_params, headers, **kwargs)
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/cosmos/_cosmos_client_connection.py", line 2249, in __Get
    return synchronized_request.SynchronizedRequest(
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/cosmos/_synchronized_request.py", line 206, in SynchronizedRequest
    return _retry_utility.Execute(
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/cosmos/_retry_utility.py", line 83, in Execute
    result = ExecuteFunction(function, global_endpoint_manager, *args, **kwargs)
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/cosmos/_retry_utility.py", line 144, in ExecuteFunction
    return function(*args, **kwargs)
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/cosmos/_synchronized_request.py", line 131, in _Request
    response = _PipelineRunFunction(
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/cosmos/_synchronized_request.py", line 172, in _PipelineRunFunction
    return pipeline_client._pipeline.run(request, **kwargs)
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 205, in run
    return first_node.send(pipeline_request)  # type: ignore
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  [Previous line repeated 1 more time]
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/cosmos/_retry_utility.py", line 215, in send
    raise err
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/cosmos/_retry_utility.py", line 192, in send
    response = self.next.send(request)
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  [Previous line repeated 1 more time]
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 100, in send
    self._sender.send(request.http_request, **request.context.options),
  File "/home/romain/.asdf/installs/python/3.10.10/lib/python3.10/site-packages/azure/core/pipeline/transport/_requests_basic.py", line 376, in send
    raise error
azure.core.exceptions.ServiceRequestError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:997)

My Cosmos DB is running fine, I can access the web console and had exported the end point and key in my env :

Is there a way to get rid of this ?

Maybe this exception could be catched with a user friendly-er message ?

seesharprun commented 1 year ago

Hello!

The emulator has a certificate that you must download from the emulator in order to use it with SSL/TLS: https://learn.microsoft.com/en-us/azure/cosmos-db/docker-emulator-linux?tabs=sql-api%2Cssl-netstd21#run-the-linux-emulator-on-linux-os. It's pretty common to spin up a new sample application and run into this error since the SSL certificate validation would fail out of the box.

I would try the troubleshooting section at the bottom of the article for tips on resolving this. If you can't resolve, feel free to reopent he issue.

Rom1deTroyes commented 1 year ago

Well, lgtm to state that I haven't read the doc, forgotten a step or doing a bad thing, so retrying step by step this morning with a brand new env :

❯ env |grep HOST_IP
HOST_IP=172.29.128.1
❯ env |grep -i cosmos
COSMOS_ENDPOINT=https://localhost:8081
COSMOS_KEY=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==

❯ curl -k https://$HOST_IP:8081/_explorer/emulator.pem > ./emulatorcert.crt && sudo mv ./emulatorcert.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates && more /etc/ssl/certs/emulatorcert.pem
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1212  100  1212    0     0   107k      0 --:--:-- --:--:-- --:--:--  107k
[sudo] password for romain:
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

done.
done.
-----BEGIN CERTIFICATE-----
MIIDQzCCAiugAwIBAgIQfsdXy6hTHZFDSc9mhpVaYTANBgkqhkiG9w0BAQUFADAU
MRIwEAYDVQQDEwlsb2NhbGhvc3QwHhcNMjMwMzIyMTAyODAxWhcNMjgwMzIxMTAy
ODAxWjAUMRIwEAYDVQQDEwlsb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQD3Ei3dPL0FPpUjPTaG3VduAfgdMiq+aG4ykeIX6i76aURfizWW
zHgNpyA7/q1eJvQrjEH4zqi1EQFAMJFH9CWYYTVNkem+IynH4/DMBGUzjB5M9gtx
ODVqXeoFDoUbWZd3mOSjpf0Zf/FqP+N7zOdD4xH7g6K0p8Vj3e73RCCp1NI6HPA1
qbPieFVEJUqjw6OsfsbJFXmNWqhxo8t9L2nJX0BDK+9SSqA6S+8Fllk8creftlpb
r6bdwTEVURLW2D1IFH8AB2j+LSXpd9jSMfDSZTyHYPCgvPC7+S2d6rOyGzWVZc2x
/YlokeCIU9+lnfrxCIio5X3DiqgCirVKsIUdAgMBAAGjgZAwgY0wewYDVR0RAQH/
BHEwb4ITOGE5YTQ0OWM3YjY0LkRPTUFJToIJbG9jYWxob3N0hwSsFAADhwR/AAAB
hwSsHYABhwSsFAADhwSsHYABggoxNzIuMjAuMC4zggkxMjcuMC4wLjGCDDE3Mi4y
OS4xMjguMYIKMTcyLjIwLjAuMzAOBgNVHQ8BAf8EBAMCAqQwDQYJKoZIhvcNAQEF
BQADggEBAGZPMgNucN/8ETcNMysdbgv+Z3AwHqZmejxLF8Nm5vU+4j3YCvriDBfH
iknNTS4bAa7GAZPz1nuhZfMDuN9P6Pkj1SIBDCSMWnjAcFyy9mGmbND0F5mYTCG4
i6JOQKp/WBRh8iRJ8z95nmJsAjFWJAkfQj0SM+33sz9zl1ZwlxuFMn68DVmAj/cf
iIgeanyUTkBC1ga9trZruoYumzIv/gTtaQSEXa0SOcJ2vJKf0+CrhNTUDZ9oAvAr
Y8uAlKPS+ult3UGJMV7sM5X0f9D5iGCWB9ErEOp0q/qhBf/82Gf4OTPdt2yEcLoS
A3VFo1To4PrdiIBBTHh5J9h424WcTcM=
-----END CERTIFICATE-----
❯ python 001-quickstart/app.py
Traceback (most recent call last):
  File "001-quickstart/app.py", line 27, in <module>
    database = client.create_database_if_not_exists(id=DATABASE_NAME)
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/cosmos/cosmos_client.py", line 319, in create_database_if_not_exists
    database_proxy.read(
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/cosmos/database.py", line 141, in read
    self._properties = self.client_connection.ReadDatabase(
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/cosmos/_cosmos_client_connection.py", line 342, in ReadDatabase
    return self.Read(path, "dbs", database_id, None, options, **kwargs)
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/cosmos/_cosmos_client_connection.py", line 2201, in Read
    result, self.last_response_headers = self.__Get(path, request_params, headers, **kwargs)
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/cosmos/_cosmos_client_connection.py", line 2249, in __Get
    return synchronized_request.SynchronizedRequest(
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/cosmos/_synchronized_request.py", line 206, in SynchronizedRequest
    return _retry_utility.Execute(
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/cosmos/_retry_utility.py", line 83, in Execute
    result = ExecuteFunction(function, global_endpoint_manager, *args, **kwargs)
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/cosmos/_retry_utility.py", line 144, in ExecuteFunction
    return function(*args, **kwargs)
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/cosmos/_synchronized_request.py", line 131, in _Request
    response = _PipelineRunFunction(
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/cosmos/_synchronized_request.py", line 172, in _PipelineRunFunction
    return pipeline_client._pipeline.run(request, **kwargs)
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 205, in run
    return first_node.send(pipeline_request)  # type: ignore
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  [Previous line repeated 1 more time]
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/cosmos/_retry_utility.py", line 215, in send
    raise err
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/cosmos/_retry_utility.py", line 192, in send
    response = self.next.send(request)
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 69, in send
    response = self.next.send(request)
  [Previous line repeated 1 more time]
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/core/pipeline/_base.py", line 100, in send
    self._sender.send(request.http_request, **request.context.options),
  File "/home/romain/.asdf/installs/python/3.8.13/lib/python3.8/site-packages/azure/core/pipeline/transport/_requests_basic.py", line 376, in send
    raise error
azure.core.exceptions.ServiceRequestError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1131)

I also tried with COSMOS_ENDPOINT=https://172.29.128.1:8081 with the same result, and verified that https://localhost:8081/_explorer/index.html doesn't complain about the certificate.

Note that :

for Node.js, you have an environment variable to deal with this, so I wonder if there is something similar for Python,
with the CosmosDB emulator, TLS verification is disabled, so it fair to expect the same behavior from the docker emulator.

Any idea to go ahead ?

Rom1deTroyes commented 1 year ago

Also, note that the 002-quickstart-async/app.py works :

❯ python 002-quickstart-async/app.py
Database         cosmicworks
Container        products
Point read       Yamba Surfboard
Result list      [
 {
  "id": "70b63682-b93a-4c77-aad2-65501347265f",
  "categoryId": "61dba35b-4f02-45c5-b648-c6badc0cbd79",
  "categoryName": "gear-surf-surfboards",
  "name": "Yamba Surfboard",
  "quantity": 12,
  "sale": false,
  "_rid": "znwuAMJNZVABAAAAAAAAAA==",
  "_self": "dbs/znwuAA==/colls/znwuAMJNZVA=/docs/znwuAMJNZVABAAAAAAAAAA==/",
  "_etag": "\"00000000-0000-0000-5e32-21efa4c601d9\"",
  "_attachments": "attachments/",
  "_ts": 1679649739
 }
]

seesharprun commented 1 year ago

I'm not as much of an expert on Python, but I did see two StackOverflow topics where someone defined a function named no_ssl_verification and then used it with the library. Does that help?

Rom1deTroyes commented 1 year ago

Defining the certificate in an environment variable solve the issue without altering the code :

export REQUESTS_CA_BUNDLE='/etc/ssl/certs/emulatorcert.pem'

Still don't understand why it worked with async queries, but I will address the azure-cosmos tracker when I'll know more about the lib :-)

Thanx for these samples, it's very helpful as boilerplate for quick starting !

Azure-Samples / cosmos-db-nosql-python-samples

Can't use self signed certificate (using Docker container for Cosmos DB) with 001-quickstart samples #12