search

Azure / iotedgehubdev

IoT Edge Hub Dev Tool
Other
88 stars 31 forks source link

Python Module Client initialization throws exception (CERTIFICATE_VERIFY_FAILED) #311

Closed perttvil closed 2 years ago

perttvil commented 3 years ago

I am developing Azure Iot Edge modules with VsCode with its extensions and iotedgehubdev with MacOs.

All Node.js modules work fine, when running them with VsCode's run in 'Build and Run Iot Edge Solution in Simulator' command.

But when I create a Python module with VsCode wizard, IoTHubModuleClient initialization doesn't work at all. And I cannot run it at all.

Following code throws an exception:

# The client object is used to interact with your Azure IoT hub.
module_client = IoTHubModuleClient.create_from_edge_environment()

Exception:

PythonModule                  | transport.connect raised error
PythonModule                  | Traceback (most recent call last):
PythonModule                  |   File "/usr/local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 340, in connect
PythonModule                  |     host=self._hostname, port=8883, keepalive=DEFAULT_KEEPALIVE
PythonModule                  |   File "/usr/local/lib/python3.7/site-packages/paho/mqtt/client.py", line 941, in connect
PythonModule                  |     return self.reconnect()
PythonModule                  |   File "/usr/local/lib/python3.7/site-packages/paho/mqtt/client.py", line 1104, in reconnect
PythonModule                  |     sock.do_handshake()
PythonModule                  |   File "/usr/local/lib/python3.7/ssl.py", line 1139, in do_handshake
PythonModule                  |     self._sslobj.do_handshake()
PythonModule                  | ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa'. (_ssl.c:1091)
PythonModule                  | 
PythonModule                  | The above exception was the direct cause of the following exception:
PythonModule                  | 
PythonModule                  | Traceback (most recent call last):
PythonModule                  |   File "/usr/local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_stages_mqtt.py", line 115, in _run_op
PythonModule                  |     self.transport.connect(password=self.sas_token)
PythonModule                  |   File "/usr/local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 345, in connect
PythonModule                  |     raise exceptions.ConnectionFailedError(cause=e)
PythonModule                  | azure.iot.device.common.transport_exceptions.ConnectionFailedError: None caused by [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa'. (_ssl.c:1091)
PythonModule                  | 
PythonModule                  | ConnectOperation: completing with error None caused by [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa'. (_ssl.c:1091)
PythonModule                  | RetryStage(ConnectOperation): Op needs retry with interval 20 because of None caused by [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa'. (_ssl.c:1091).  Setting timer.

I have not installed or changed any certificates in my local dev environment and I don't get any errors from NodeJs modules.

Api doc says: https://docs.microsoft.com/en-us/python/api/azure-iot-device/azure.iot.device.iothubmoduleclient?view=azure-python#create-from-edge-environment---kwargs-

This method can only be run from inside an IoT Edge container, or in a debugging environment configured for Edge development (e.g. Visual Studio, Visual Studio Code)

Does that include iotedgehubdev? And if not, how am I supposed to run in simulator?

Code can be found from here: https://github.com/perttvil/azure-iot-edge-python-module

mhshami01 commented 3 years ago

@perttvil - Thank you for your input. We have a couple of questions:

perttvil commented 3 years ago
  • does the module fail on the device also or only in the simulator?

Just tested and the module works just fine on real device without any changes.

  • does the issue show up with a vanilla module?

Yes it does. Issue does not go away, even though I deleted all unnecessary parts away.

mhshami01 commented 3 years ago

Hello @perttvil,

Using the steps from Use Visual Studio Code to develop and debug modules for Azure IoT Edge, I was able to use IoTEdgeHubDev to debug a Python module. The article covers several options for languages and module types.

Sincerely, --Haitham

perttvil commented 3 years ago

Using the steps from Use Visual Studio Code to develop and debug modules for Azure IoT Edge, I was able to use IoTEdgeHubDev to debug a Python module. The article covers several options for languages and module types.

Yes, I am able to do that also. But that is not the problem I am talking about.

Problem is that Visual Studio Code Build and Run Iot Edge Solution in Simulator does not work when Python module in involved. It works for me with NodeJs and C# modules. But not for Python.

And python fails to get ModuleClient because of certificate validation errors like I mentioned in the initial post in this thread.

To test: Right click the deployment.template.json file in Visual Studio Code. Select: Build and Run Iot Edge Solution in Simulator and you will see the error.

marianan commented 3 years ago

Yes, I am able to do that also. But that is not the problem I am talking about.

@perttvil thanks for confirming that the default Python module that is generated from VS Code runs fine in the simulator. Your deployment manifest appears identical to the default - what are the changes in your module that trigger the issue?

A couple more things to try:

  1. which version of python are running with? The simulator runs on 2.7 up to 3.8
  2. delete all docker containers and restart docker
perttvil commented 3 years ago

@perttvil thanks for confirming that the default Python module that is generated from VS Code runs fine in the simulator. Your deployment manifest appears identical to the default - what are the changes in your module that trigger the issue?

Only debugging single module works, but that is not my problem.

Can you confirm to me that you can actually execute Python module As a Solution with the simulator?

I think in that case PythonModule creates the ModuleClient to a different endpoint or in different way, and connection certificate fails when running as a Solution. (It works when running a single Module).

In Visual Code the action Build and Run Iot Edge Solution in Simulator is the one that does not work.

The command that Visual Studio Code generates is: docker build --rm -f "/opt/PythonSkeleton/modules/PythonModule/Dockerfile.amd64" -t localhost:5000/pythonmodule:0.0.1-amd64 "/opt/PythonSkeleton/modules/PythonModule" && sudo "iotedgehubdev" start -d "/opt/PythonSkeleton/config/deployment.amd64.json" -v

I am running on MacOs 11.2.3.

python --version
Python 2.7.16
python3 --version
Python 3.9.1
iotedgehubdev --version
iotedgehubdev, version 0.14.3

Python inside the PythonModule docker container is 3.7.

Generating docker containers did not help.

The whole stacktrace is here, and there seems to be an certificate problem with PythonModule and edgeHubDev.

PythonModule                  | transport.connect raised error
PythonModule                  | Traceback (most recent call last):
PythonModule                  |   File "/usr/local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 340, in connect
PythonModule                  |     host=self._hostname, port=8883, keepalive=DEFAULT_KEEPALIVE
PythonModule                  |   File "/usr/local/lib/python3.7/site-packages/paho/mqtt/client.py", line 941, in connect
PythonModule                  |     return self.reconnect()
PythonModule                  |   File "/usr/local/lib/python3.7/site-packages/paho/mqtt/client.py", line 1104, in reconnect
PythonModule                  |     sock.do_handshake()
PythonModule                  |   File "/usr/local/lib/python3.7/ssl.py", line 1139, in do_handshake
PythonModule                  |     self._sslobj.do_handshake()
PythonModule                  | ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa'. (_ssl.c:1091)
PythonModule                  | 
PythonModule                  | The above exception was the direct cause of the following exception:
PythonModule                  | 
PythonModule                  | Traceback (most recent call last):
PythonModule                  |   File "/usr/local/lib/python3.7/site-packages/azure/iot/device/common/pipeline/pipeline_stages_mqtt.py", line 115, in _run_op
PythonModule                  |     self.transport.connect(password=self.sas_token)
PythonModule                  |   File "/usr/local/lib/python3.7/site-packages/azure/iot/device/common/mqtt_transport.py", line 345, in connect
PythonModule                  |     raise exceptions.ConnectionFailedError(cause=e)
PythonModule                  | azure.iot.device.common.transport_exceptions.ConnectionFailedError: None caused by [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa'. (_ssl.c:1091)
PythonModule                  | 
PythonModule                  | ConnectOperation: completing with error None caused by [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa'. (_ssl.c:1091)
PythonModule                  | RetryStage(ConnectOperation): Op needs retry with interval 20 because of None caused by [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa'. (_ssl.c:1091).  Setting timer.
edgeHubDev                    | <4> 2021-03-17 10:45:17.894 +00:00 [WRN] - TLS handshake failed., System.AggregateException: One or more errors occurred. (Authentication failed, see inner exception.)
edgeHubDev                    |  ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
edgeHubDev                    |  ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
edgeHubDev                    |  ---> Interop+Crypto+OpenSslCryptographicException: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
edgeHubDev                    |    --- End of inner exception stack trace ---
edgeHubDev                    |    at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, Byte[] recvBuf, Int32 recvOffset, Int32 recvCount, Byte[]& sendBuf, Int32& sendCount)
edgeHubDev                    |    at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteContext& context, ArraySegment`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
edgeHubDev                    |    --- End of inner exception stack trace ---
edgeHubDev                    |    at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
edgeHubDev                    |    at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
edgeHubDev                    |    at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
edgeHubDev                    |    at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
edgeHubDev                    |    at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
edgeHubDev                    |    at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
edgeHubDev                    | --- End of stack trace from previous location where exception was thrown ---
edgeHubDev                    |    at System.Net.Security.SslStream.ThrowIfExceptional()
edgeHubDev                    |    at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
edgeHubDev                    |    at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
edgeHubDev                    |    at System.Net.Security.SslStream.EndAuthenticateAsServer(IAsyncResult asyncResult)
edgeHubDev                    |    at System.Net.Security.SslStream.<>c.<AuthenticateAsServerAsync>b__68_2(IAsyncResult iar)
edgeHubDev                    |    at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
edgeHubDev                    |    --- End of inner exception stack trace ---, 272879b9
perttvil commented 3 years ago

And I changed my laptop python3 in $PATH from 3.9 to 3.8, which did not help, since iotedgehubdev python script seems to select python 3.7 by default, which I also have installed.

marianan commented 3 years ago

I am confirming that Python modules execute successfully in both single module and solution modes. Could you execute these steps and share your findings:

  1. Close all instances of VSCode and open a new one
  2. Restart docker and make sure no containers are running
  3. Create a new Python module without code or config modifications
  4. Right click on deployment.template.json and select "Build and run IoT Edge solution in the simulator"
  5. After build has completed and simulator has started or failed to start, run docker ps -a and capture output

There is a cert helper container that sets up the necessary certs, I wonder if it is there on your system. Here is the list of my docker containers after the above steps for reference: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES fc2bffd06e9c mcr.microsoft.com/azureiotedge-simulated-temperature-sensor:1.0 "/bin/sh -c 'echo \"$…" 3 minutes ago Up 3 minutes SimulatedTemperatureSensor efb99769f00e localhost:5000/samplemodule:0.0.1-amd64 "python3 -u ./main.py" 3 minutes ago Up 3 minutes SampleModule 71418ee629b5 mcr.microsoft.com/azureiotedge-hub:1.0 "/bin/sh -c 'echo \"$…" 4 minutes ago Up 3 minutes 0.0.0.0:443->443/tcp, 0.0.0.0:5671->5671/tcp, 0.0.0.0:8883->8883/tcp edgeHubDev 4d0283ac7ca3 hello-world:latest "/hello" 4 minutes ago Created cert_helper a0d15e0db748 registry:2 "/entrypoint.sh /etc…" 4 minutes ago Up 4 minutes 0.0.0.0:5000->5000/tcp registry

perttvil commented 3 years ago

Still does not work. Same certificate validation error and python module (SampleModule) keeps restarting since it fails to create a ModuleClient.

I do not have a cert_helper container running, nor can I find an image, which has a name anything like that. So I guess I don't have any cert helper running locally.

Here is the list of containers (docker ps -a): CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 14eefaeeddbe mcr.microsoft.com/azureiotedge-simulated-temperature-sensor:1.0 "/bin/sh -c 'echo \"$…" 6 seconds ago Up 6 seconds SimulatedTemperatureSensor fb6a6fb8b6db localhost:5000/samplemodule:0.0.1-amd64 "python3 -u ./main.py" 6 seconds ago Up 6 seconds SampleModule 58528d6f7e5e mcr.microsoft.com/azureiotedge-hub:1.0 "/bin/sh -c 'echo \"$…" 7 seconds ago Up 6 seconds 0.0.0.0:443->443/tcp, 0.0.0.0:5671->5671/tcp, 0.0.0.0:8883->8883/tcp edgeHubDev 2a6553c5fdc1 hello-world:latest "/hello" 10 seconds ago Created cert_helper 8621d678c075 registry:2 "/entrypoint.sh /etc…" 2 minutes ago Up 2 minutes 0.0.0.0:5000->5000/tcp registry

Rumpkie commented 3 years ago

@perttvil, I have the same issue on my mac as well and I have not been able to resolve the issue either.

I tried clearing the device certificates and using iotedgehubdev generatedeviceca to no effect.

I have verified that the certificate that is generated is the one being used inside the container.

It also works fine for the nodejs module and temperature sensor module too.

I should add if it helps anybody that testing the link running a

openssl s_client -connect edgehubdev:8883 -showcerts -CAfile $EdgeModuleCACertificateFile from inside the container

yields a 'Verify return code: 0 (ok)' on the SSL connection using TLS 1.2

I have gone so far as blowing way vs code, python, and docker and starting over to no avail.

Rumpkie commented 3 years ago

I also want to add that I have tested this against azure-iot-device version 2.0.0 through 2.5.1

Rumpkie commented 3 years ago

Still have not been able to resolve this issue.

marianan commented 3 years ago

This may be Mac specific issue. We will test on Mac and confirm if we can repro locally.

marianan commented 2 years ago

@perttvil the symptoms you are observing appear the same as https://github.com/Azure/iotedgehubdev/issues/308. Please try to develop and debug using the VSCode devcontainers instead of native MacOS env and let us know if you are still blocked. For more details on the root cause of what you are observing currently (this applies to Python SDK too) - https://github.com/Azure/iotedgehubdev/issues/308#issuecomment-949966939

marianan commented 2 years ago

Closing this thread, please reach back for any questions.

bedzinsk commented 2 years ago

I have exactly the same issue. Using Mac OS X. Same exceptions happens when running iotedgehubdev locally and identical error when running from Visual Studio Code:

sample_module    | Traceback (most recent call last):
sample_module    |   File "/opt/venv/lib/python3.9/site-packages/azure/iot/device/common/mqtt_transport.py", line 390, in connect
sample_module    |     rc = self._mqtt_client.connect(
sample_module    |   File "/opt/venv/lib/python3.9/site-packages/paho/mqtt/client.py", line 914, in connect
sample_module    |     return self.reconnect()
sample_module    |   File "/opt/venv/lib/python3.9/site-packages/paho/mqtt/client.py", line 1073, in reconnect
sample_module    |     sock.do_handshake()
sample_module    |   File "/usr/lib/python3.9/ssl.py", line 1309, in do_handshake
sample_module    |     self._sslobj.do_handshake()
sample_module    | ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '37.1.168.192.in-addr.arpa'. (_ssl.c:1129)