IOT Edge for Linux on Windows (eflow) - nested VM deployement - proxy settings - corporate firewall

[usman-bin-imran]

Greetings Eflow team,

My team is working on achieving following:

1. Sending telemetry data from OPC Server to Azure Iot Hub.
2. The environment has restricted internet, so we opened required urls and ports (outbound).
3. OPC Server has no internet, so we made a Windows Virtual Machine on same environment.
4. This Windows Virtual Machine has IoT Edge for Linux on Windows (EFLOW) deployed on it.
5. We planned to send telemetry data using OPC Publisher Module to IoT Hub.

Helping Microsoft Article: https://learn.microsoft.com/en-us/azure/iot-edge/how-to-configure-proxy-support?view=iotedge-1.4

The planned architecture is:

Unfortunately, we are experiencing issues while achieving above. I am attaching logs. Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:26Z [INFO] - Starting Azure IoT Edge Daemon Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:26Z [INFO] - Version - 1.4.20 Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:26Z [INFO] - Obtaining Edge device provisioning data... Dec 26 18:19:26 WINDOWSVM-EFLOW systemd[1]: Started Azure IoT Identity Service. Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Starting service... Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Version - dev build Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Detected HTTPS proxy server http://123.45.67.89:8080/ Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Provisioning starting. Reason: Startup Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Updated device info for myEdgeDevice. Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Provisioning complete. Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Identity reconciliation started. Reason: Startup Dec 26 18:19:26 WINDOWSVM-EFLOW systemd[1]: Started Azure IoT Keys Service. Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - Starting service... Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - Version - dev build Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - Loaded libaziot-keys with version 0x02010000 Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - Starting server... Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"} Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "402"} Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: unexpected EOF Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: unexpected EOF Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [WARN] - Network not available for Identity reconciliation. Using offline backup from last run. Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [INFO] - Starting server... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [INFO] - <-- POST /identities/device?api-version=2020-09-01 {"content-type": "application/json", "host": "identityd.sock", "content-length": "16"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Device is myEdgeDevice on myIOThub.azure-devices.net Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Initializing module runtime... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Using runtime network id azure-iot-edge Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Successfully initialized module runtime Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - <-- POST /keypair?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "61"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:30 WINDOWSVM-EFLOW systemd[1]: Started Azure IoT Certificates Service. Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - Starting service... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - Version - dev build Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - Starting server... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - <-- GET /certificates/aziot-edged-ca?api-version=2020-09-01 {"host": "certd.sock"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Using existing Edge CA certificate Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - <-- GET /certificates/aziot-edged-ca?api-version=2020-09-01 {"host": "certd.sock"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Certificate aziot-edged-ca will be auto-renewed. Next renewal at 2024-03-06T13:02:07+00:00. Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Stopping all modules... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - All modules stopped Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Detecting if device information has changed... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Device information has not changed Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Starting management API... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Starting workload API... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Starting image garbage collection task... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Starting watchdog with 60 second period... Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Watchdog checking Edge runtime status Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [INFO] - <-- PUT /identities/modules/$edgeAgent?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": "identityd.sock", "content-length": "40"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "402"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: unexpected EOF Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:34Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: unexpected EOF Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:34Z [INFO] - !!! Hub client error Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:34Z [INFO] - !!! caused by: error trying to connect: unexpected EOF Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:34Z [INFO] - !!! caused by: unexpected EOF Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:34Z [INFO] - --> 404 {"content-type": "application/json"} Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:34Z [WARN] - Error in watchdog: Failed to update $edgeAgent identity: Hub client error Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-edged[14160]: caused by: error trying to connect: unexpected EOF Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-edged[14160]: caused by: unexpected EOF Dec 26 18:19:35 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:35Z [INFO] - <-- POST /identities/device/reprovision?api-version=2020-09-01 {"content-type": "application/json", "host": "identityd.sock", "content-length": "16"} Dec 26 18:19:35 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:35Z [INFO] - Provisioning starting. Reason: Api Dec 26 18:19:35 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:35Z [INFO] - Updated device info for myEdgeDevice. Dec 26 18:19:35 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:35Z [INFO] - Provisioning complete. Dec 26 18:19:35 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:35Z [INFO] - Identity reconciliation started. Reason: Api Dec 26 18:19:36 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:36Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"} Dec 26 18:19:36 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:36Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:36 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:36Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "402"} Dec 26 18:19:36 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:36Z [INFO] - --> 200 {"content-type": "application/json"} Dec 26 18:19:36 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:36Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: unexpected EOF Dec 26 18:19:39 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:39Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: unexpected EOF Dec 26 18:19:39 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:39Z [INFO] - !!! Hub client error Dec 26 18:19:39 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:39Z [INFO] - !!! caused by: error trying to connect: unexpected EOF Dec 26 18:19:39 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:39Z [INFO] - !!! caused by: unexpected EOF Dec 26 18:19:39 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:39Z [INFO] - --> 404 {"content-type": "application/json"}

[usman-bin-imran]

Output when I executed: sudo iotedge check

Configuration checks (aziot-identity-service)

√ keyd configuration is well-formed - OK √ certd configuration is well-formed - OK √ tpmd configuration is well-formed - OK √ identityd configuration is well-formed - OK √ daemon configurations up-to-date with config.toml - OK √ identityd config toml file specifies a valid hostname - OK × aziot-identity-service package is up-to-date - Error could not query https://aka.ms/latest-aziot-identity-service for latest available version ‼ host time is close to reference time - Warning Could not query NTP server √ preloaded certificates are valid - OK √ keyd is running - OK √ certd is running - OK √ identityd is running - OK √ read all preloaded certificates from the Certificates Service - OK √ read all preloaded key pairs from the Keys Service - OK √ check all EST server URLs utilize HTTPS - OK √ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)

× host can connect to and perform TLS handshake with iothub AMQP port - Error Could not connect to myIOThub.azure-devices.net : could not complete TLS handshake × host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Error Could not connect to myIOThub.azure-devices.net : could not complete TLS handshake × host can connect to and perform TLS handshake with iothub MQTT port - Error Could not connect to myIOThub.azure-devices.net : could not complete TLS handshake

Configuration checks

√ aziot-edged configuration is well-formed - OK √ configuration up-to-date with config.toml - OK √ container engine is installed and functional - OK × configuration has correct URIs for daemon mgmt endpoint - Error Unable to find image 'mcr.microsoft.com/azureiotedge-diagnostics:1.4.20' locally docker: Error response from daemon: Get "https://mcr.microsoft.com/v2/": tls: failed to verify certificate: x509: certificate signed by unknown authority. See 'docker run --help'. × aziot-edge package is up-to-date - Error Error while fetching latest versions of edge components: could not send HTTP request × container time is close to host time - Error Could not query local time inside container ‼ DNS server - Warning Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub. Please see https://aka.ms/iotedge-prod-checklist-dns for best practices. You can ignore this warning if you are setting DNS server per module in the Edge deployment. ‼ production readiness: logs policy - Warning Container engine is not configured to rotate module logs which may cause it run out of disk space. Please see https://aka.ms/iotedge-prod-checklist-logs for best practices. You can ignore this warning if you are setting log policy per module in the Edge deployment. × production readiness: Edge Agent's storage directory is persisted on the host filesystem - Error Could not check current state of edgeAgent container × production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error Could not check current state of edgeHub container × Agent image is valid and can be pulled from upstream - Error Failed to get edge Agent image ‼ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - Warning The proxy setting for IoT Edge Agent "", IoT Edge Daemon "http://123.45.67.89:8080", IoT Identity Daemon "http://123.45.67.89:8080", and Moby "" may need to be identical.

Connectivity checks

× container on the default network can connect to upstream AMQP port - Error Container on the default network could not connect to myIOThub.azure-devices.net:5671 × container on the default network can connect to upstream HTTPS / WebSockets port - Error Container on the default network could not connect to myIOThub.azure-devices.net:443 × container on the IoT Edge module network can connect to upstream AMQP port - Error Container on the azure-iot-edge network could not connect to myIOThub.azure-devices.net:5671 × container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - Error Container on the azure-iot-edge network could not connect to myIOThub.azure-devices.net:443 17 check(s) succeeded. 4 check(s) raised warnings. Re-run with --verbose for more details. 14 check(s) raised errors. Re-run with --verbose for more details. 2 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

[gordonwang0]

Your device is unable to connect to IoT Hub, likely due to an error in proxy configuration. On your device, could you check the output of the command:

curl -v -x http://123.45.67.89:8080/ \
    https://testiothub114.azure-devices.net/devices/myIOThub.azure-devices.net/devices/myEdgeDevice/modules?api-version=2017-11-08-preview

You should get an HTTP 401 Unauthorized response, but it should be able to connect.

[usman-bin-imran]

@gordonwang0

Thanks for looking into the matter. Execution result of curl -v -x proxy iot-edge-device is:

[gordonwang0]

That's not the expected output for a correctly-configured proxy. You'll have to debug your proxy configuration. Check that connections to your IoT Hub are allowed and that TLS certificates are configured correctly if this proxy is intercepting TLS traffic.

[jlian]

Any updates @usman-bin-imran

[usman-bin-imran]

Hi @gordonwang0 @vadim-kovalyov @jlian

Thankyou for the attention to the matter. I am experiencing problems while debugging. Can you give a pathway that I can follow, and see if TLS certificates are properly configured? As, the outbound urls and ports for IoT Hub are already allowed.

Looking forward,

Usman Bin Imran

[vadim-kovalyov]

@usman-bin-imran, the only thing I see and can suggest to look into is to investigate the TLS error - curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL.

Make sure that your system supports minimum required version of TLS (1.2) and you have proper root certs installed https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-tls-support

[jlian]

@usman-bin-imran any updates?

[usman-bin-imran]

@jlian, we forwarded the request to our organisation's networking department to reconfigure proxy settings properly. Please keep the issue thread active untill we get back to you.

Thanks for looking into the matter!

Kr, Usman Bin Imran

[usman-bin-imran]

Hi @gordonwang0 @vadim-kovalyov @jlian

Thanks for staying in loop. The proxy has ben set successfully by our networking department. Now we are able to send messages to IoT Hub using python code. While trying to re-communicate IoT Edge for Linux on Windows (EFLOW) with IoT Hub through proxy, we are still encountering errors below (after python code):

------------Python Code Starts Here------------ from azure.iot.device import ProxyOptions, IoTHubDeviceClient, Message os.environ['HTTP_PROXY'] = 'http://123.45.67.89:8080' os.environ['HTTPS_PROXY'] = 'http://123.45.67.89:8080' proxy_opts = ProxyOptions( proxy_type="HTTP", proxy_addr="123.45.67.89", proxy_port=8080, proxy_username=None, proxy_password=None) device_client = IoTHubDeviceClient.create_from_connection_string(connection_string = iot_hub_conn_str, proxy_options = proxy_opts, websockets=True) message = Message(str(temporaryValue)) device_client.send_message(message) ------------Python Code Ends Here------------

------------sudo iotedge check Output Starts Here------------ PS C:\WINDOWS\system32> Connect-EflowVm iotedge-user@WINDOWS01-EFLOW [ ~ ]$ sudo iotedge check

Configuration checks (aziot-identity-service)

√ keyd configuration is well-formed - OK √ certd configuration is well-formed - OK √ tpmd configuration is well-formed - OK √ identityd configuration is well-formed - OK √ daemon configurations up-to-date with config.toml - OK √ identityd config toml file specifies a valid hostname - OK × aziot-identity-service package is up-to-date - Error could not query https://aka.ms/latest-aziot-identity-service for latest available version ‼ host time is close to reference time - Warning Could not query NTP server √ preloaded certificates are valid - OK √ keyd is running - OK √ certd is running - OK √ identityd is running - OK √ read all preloaded certificates from the Certificates Service - OK √ read all preloaded key pairs from the Keys Service - OK √ check all EST server URLs utilize HTTPS - OK √ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)

× host can connect to and perform TLS handshake with iothub AMQP port - Error Could not connect to myIoTHub.azure-devices.net : could not complete TLS handshake × host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Error Could not connect to myIoTHub.azure-devices.net : could not complete TLS handshake × host can connect to and perform TLS handshake with iothub MQTT port - Error Could not connect to myIoTHub.azure-devices.net : could not complete TLS handshake

Configuration checks

√ aziot-edged configuration is well-formed - OK √ configuration up-to-date with config.toml - OK √ container engine is installed and functional - OK √ configuration has correct URIs for daemon mgmt endpoint - OK × aziot-edge package is up-to-date - Error Error while fetching latest versions of edge components: could not send HTTP request √ container time is close to host time - OK √ DNS server - OK √ production readiness: logs policy - OK × production readiness: Edge Agent's storage directory is persisted on the host filesystem - Error Could not check current state of edgeAgent container × production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error Could not check current state of edgeHub container √ Agent image is valid and can be pulled from upstream - OK ‼ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - Warning The proxy setting for IoT Edge Agent "", IoT Edge Daemon "https://123.45.67.89:8080", IoT Identity Daemon "https://123.45.67.89:8080", and Moby "" may need to be identical.

Connectivity checks

× container on the default network can connect to upstream AMQP port - Error Container on the default network could not connect to myIoTHub.azure-devices.net:5671 × container on the default network can connect to upstream HTTPS / WebSockets port - Error Container on the default network could not connect to myIoTHub.azure-devices.net:443 × container on the IoT Edge module network can connect to upstream AMQP port - Error Container on the azure-iot-edge network could not connect to myIoTHub.azure-devices.net:5671 × container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - Error Container on the azure-iot-edge network could not connect to myIoTHub.azure-devices.net:443 22 check(s) succeeded. 2 check(s) raised warnings. Re-run with --verbose for more details. 11 check(s) raised errors. Re-run with --verbose for more details. 2 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details. ------------sudo iotedge check Output Ends Here------------

------------cURL check suggested by @gordonwang0 ------------

I also re-provisioned the device and restarted it several times. Really appreciate your attention to the matter!

Looking forward,

Usman Bin Imran +92-310-480-160-1

[jlian]

@Azure/iotedge-eflow any ideas?

@vadim-kovalyov do you think this might be due to Moby proxy settings misconfiguration, given the check result

‼ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - Warning The proxy setting for IoT Edge Agent "", IoT Edge Daemon "https://123.45.67.89:8080/", IoT Identity Daemon "https://123.45.67.89:8080/", and Moby "" may need to be identical.

[vadim-kovalyov]

Hey, I think at least two warnings need to be addressed first:

‼ host time is close to reference time - Warning ‼ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - Warning The proxy setting for IoT Edge Agent "", IoT Edge Daemon "https://123.45.67.89:8080/", IoT Identity Daemon "https://123.45.67.89:8080/", and Moby "" may need to be identical.

As you can see, Edge Agent and Moby settings are incorrect (empty string), that's why EA can't connect and/or pull any container.

[usman-bin-imran]

Hi @gordonwang0 @vadim-kovalyov @jlian @PatAltimore

Thanks for staying in loop. I am able to send messages to IoT Hub through proxy server using Custom Python Code. I am curious that IoT Edge for Linux on Windows (EFLOW) is not working properly in nested VM situation when outbounds need to be sent through proxy. In response to @vadim-kovalyov previous response, I rechecked the applied configurations for Moby Daemon. Please review my findings below:

On the Microsoft Document: https://learn.microsoft.com/en-us/azure/iot-edge/how-to-configure-proxy-support?view=iotedge-1.4#moby-daemon

I selected Configure docker daemon on Linux. This took me to the following link: https://docs.docker.com/config/daemon/systemd/#httphttps-proxy

In the Docker Daemon Configuration part, it suggested that: "If you're behind an HTTP or HTTPS proxy server, for example in corporate settings, the daemon proxy configurations must be specified in the systemd service file, not in the daemon.json file or using environment variables."

I followed the following steps because, I am behind proxy server (corporate settings):

This is how my "/etc/systemd/system/docker.service.d/http-proxy.conf" file looks like:

Finally, it suggested to run following commands: sudo systemctl daemon-reload sudo systemctl restart docker sudo systemctl show --property=Environment docker

Output: Environment=HTTP_PROXY=http://123.45.67.89:8080 HTTPS_PROXY=https://123.45.67.89:8080

I think that "Moby Daemon" part on the microsoft document is not addressed properly for IoT Edge for Linux on Windows (nested VM environment). EFLOW is not able to read set configurations from "/etc/systemd/system/docker.service.d/http-proxy.conf".

Looking forward to hearing from you!

Kind regards, Usman Bin Imran +92-310-480-160-1

[usman-bin-imran]

Hi @gordonwang0 @vadim-kovalyov @jlian @PatAltimore @ksaye

The latest sudo iotedge check using proxy gives following results:

root@WINDOWSVM-EFLOW [ ~ ]# sudo iotedge check --proxy-uri http://123.45.67.89:8080 Configuration checks (aziot-identity-service)

√ keyd configuration is well-formed - OK √ certd configuration is well-formed - OK √ tpmd configuration is well-formed - OK √ identityd configuration is well-formed - OK √ daemon configurations up-to-date with config.toml - OK √ identityd config toml file specifies a valid hostname - OK × aziot-identity-service package is up-to-date - Error could not query https://aka.ms/latest-aziot-identity-service for latest available version ‼ host time is close to reference time - Warning Could not query NTP server √ preloaded certificates are valid - OK √ keyd is running - OK √ certd is running - OK √ identityd is running - OK √ read all preloaded certificates from the Certificates Service - OK √ read all preloaded key pairs from the Keys Service - OK √ check all EST server URLs utilize HTTPS - OK √ ensure all preloaded certificates match preloaded private keys with the same ID - OK

Connectivity checks (aziot-identity-service)

× host can connect to and perform TLS handshake with iothub AMQP port - Error Could not connect to myIOThub.azure-devices.net : could not complete TLS handshake √ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK × host can connect to and perform TLS handshake with iothub MQTT port - Error Could not connect to myIOThub.azure-devices.net : could not complete TLS handshake

Configuration checks

√ aziot-edged configuration is well-formed - OK √ configuration up-to-date with config.toml - OK √ container engine is installed and functional - OK √ configuration has correct URIs for daemon mgmt endpoint - OK × aziot-edge package is up-to-date - Error Error while fetching latest versions of edge components: could not send HTTP request √ container time is close to host time - OK √ DNS server - OK √ production readiness: logs policy - OK × production readiness: Edge Agent's storage directory is persisted on the host filesystem - Error Could not check current state of edgeAgent container × production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error Could not check current state of edgeHub container × Agent image is valid and can be pulled from upstream - Error Failed to get edge Agent image ‼ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - Warning The proxy setting for IoT Edge Agent "", IoT Edge Daemon "https://123.45.67.89:8080", IoT Identity Daemon "https://123.45.67.89:8080", and Moby "" may need to be identical.

Connectivity checks

× container on the default network can connect to upstream AMQP port - Error Container on the default network could not connect to myIOThub.azure-devices.net:5671 √ container on the default network can connect to upstream HTTPS / WebSockets port - OK × container on the IoT Edge module network can connect to upstream AMQP port - Error Container on the azure-iot-edge network could not connect to myIOThub.azure-devices.net:5671 √ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - OK 24 check(s) succeeded. 2 check(s) raised warnings. Re-run with --verbose for more details. 9 check(s) raised errors. Re-run with --verbose for more details. 2 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details.

@PatAltimore and @ksaye, Can you also please look into the matter?

Looking forward,

Usman Bin Imran +92-310-480-160-1

[jlian]

I think that "Moby Daemon" part on the microsoft document is not addressed properly for IoT Edge for Linux on Windows (nested VM environment). EFLOW is not able to read set configurations from "/etc/systemd/system/docker.service.d/http-proxy.conf".

@Azure/iotedge-eflow and @jagadishmurugan can you help here? Looks like @usman-bin-imran followed the as the docs instructed, which is to Connect-EflowVm and then apply Moby settings as per Linux instructions, but somehow it doesn't seem to be applied as shown in the latest check output.

[jagadishmurugan]

@usman-bin-imran , can you confirm you followed the instructions here https://github.com/terrymandin/EFLOWNestedEdge and specifically this part:

• Open Ports
  1. On the top device open the following ports sudo iptables -A INPUT -p tcp --dport 5671 -j ACCEPT sudo iptables -A INPUT -p tcp --dport 8883 -j ACCEPT sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT
  2. To allow pings on the top device allow ICMP messages sudo iptables -A INPUT -p icmp --icmp-type 8 -s 0/0 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
  3. To make the simulation module work on the lower device open the AMQP port sudo iptables -A INPUT -p tcp --dport 5671 -j ACCEPT

[konichi3]

@usman-bin-imran can you follow up on Jagadish's question above?

[konichi3]

@usman-bin-imran Have you had a chance to look at the question above?

[usman-bin-imran]

Hey @konichi3,

Hope this message finds you in good health. Given the limitations of the initial documentation provided by Microsoft for IoT Edge for Linux on Windows (EFLOW), as outlined in the document (https://learn.microsoft.com/en-us/azure/iot-edge/how-to-configure-proxy-support?view=iotedge-1.4), which lacks comprehensive and detailed configuration steps, we have made the decision to implement the transmission of telemetry data from the OPC Server to the IoT Hub programmatically. The solution steps in initial document were incomplete for nested-VM setup. We anticipate @Microsoft & @Azure to release an official document detailing steps, addressing the issues highlighted in the aforementioned thread.

Gratitude to @jagadishmurugan, @vadim-kovalyov, @jlian, and @gordonwang0 for staying in the loop! A special acknowledgment to @TerryWarwick for engaging with the core team!

Kind regards, Usman Bin Imran +92-310-480-160-1

[david-emakenemi]

Thank you for your feedback, @usman-bin-imran . We appreciate your input and will take it into account as we strive to improve our documentation. Our goal is to make it easier to follow. Since they are no more issues, I'll resolve this thread. Please feel free to re-open if you encounter any issues.