Azure / iotedge

The IoT Edge OSS project
MIT License
1.47k stars 462 forks source link

Proxy support https:// #7091

Closed frankvdbh closed 1 year ago

frankvdbh commented 1 year ago

Expected Behavior

We would like to limit outgoing connectivity of IoT Edge devices to a limited set of IP's as this will need to be whitelisted by customers where do not control the network. To prevent having to customer to update known Azure address space blocks in their firewall, we had the idea of sending all IoT Edge traffic through a proxy server. We followed the instructions at https://learn.microsoft.com/en-us/azure/iot-edge/how-to-configure-proxy-support?view=iotedge-1.4 But as soon as we configure the proxy to e.g. https://some-proxy-fqdn the edge Agent is no longer able to contact IoT Hub. We wonder if it is possible to use a proxy using the https:// protocol ? Because if not it makes no sense to further investigate in this solution.

Current Behavior

[WRN] [Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.BackupConfigSource] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

  1. Use an 'external' proxy service that requires https for the proxy protocol
  2. Configure edge, edgeHub, Moby etc all to use this proxy

Context (Environment)

Output of iotedge check

Click here ``` Configuration checks (aziot-identity-service) --------------------------------------------- √ keyd configuration is well-formed - OK √ certd configuration is well-formed - OK √ tpmd configuration is well-formed - OK √ identityd configuration is well-formed - OK √ daemon configurations up-to-date with config.toml - OK √ identityd config toml file specifies a valid hostname - OK √ aziot-identity-service package is up-to-date - OK √ host time is close to reference time - OK √ preloaded certificates are valid - OK √ keyd is running - OK √ certd is running - OK √ identityd is running - OK √ read all preloaded certificates from the Certificates Service - OK √ read all preloaded key pairs from the Keys Service - OK √ check all EST server URLs utilize HTTPS - OK √ ensure all preloaded certificates match preloaded private keys with the same ID - OK Connectivity checks (aziot-identity-service) -------------------------------------------- ‼ host can connect to and perform TLS handshake with iothub AMQP port - Warning Could not retrieve iothub_hostname from provisioning file. Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information. Since no hostname is provided, all hub connectivity tests will be skipped. ‼ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - Warning Could not retrieve iothub_hostname from provisioning file. Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information. Since no hostname is provided, all hub connectivity tests will be skipped. ‼ host can connect to and perform TLS handshake with iothub MQTT port - Warning Could not retrieve iothub_hostname from provisioning file. Please specify the backing IoT Hub name using --iothub-hostname switch if you have that information. Since no hostname is provided, all hub connectivity tests will be skipped. √ host can connect to and perform TLS handshake with DPS endpoint - OK Configuration checks -------------------- √ aziot-edged configuration is well-formed - OK √ configuration up-to-date with config.toml - OK √ container engine is installed and functional - OK √ configuration has correct URIs for daemon mgmt endpoint - OK √ aziot-edge package is up-to-date - OK √ container time is close to host time - OK √ DNS server - OK ‼ production readiness: logs policy - Warning Container engine is not configured to rotate module logs which may cause it run out of disk space. Please see https://aka.ms/iotedge-prod-checklist-logs for best practices. You can ignore this warning if you are setting log policy per module in the Edge deployment. ‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem. Data might be lost if the module is deleted or updated. Please see https://aka.ms/iotedge-storage-host for best practices. ‼ production readiness: Edge Hub's storage directory is persisted on the host filesystem - Warning The edgeHub module is not configured to persist its /tmp/edgeHub directory on the host filesystem. Data might be lost if the module is deleted or updated. Please see https://aka.ms/iotedge-storage-host for best practices. ‼ proxy settings are consistent in aziot-edged, aziot-identityd, moby daemon and config.toml - Warning The proxy setting for IoT Edge Agent "https://proxytest.com", IoT Edge Daemon "https://proxytest.com", IoT Identity Daemon "https://proxytest.com", and Moby "" may need to be identical. Connectivity checks ------------------- 24 check(s) succeeded. 7 check(s) raised warnings. Re-run with --verbose for more details. 7 check(s) were skipped due to errors from other checks. Re-run with --verbose for more details. Note: Replaced actual proxy address with proxytest.com as placeholder. ```

Device Information

Runtime Versions

Note: when using Windows containers on Windows, run docker -H npipe:////./pipe/iotedge_moby_engine version instead

Logs

edge-agent logs ``` <7> 2023-08-25 13:06:56.400 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Finished reconcile operation <7> 2023-08-25 13:07:01.400 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Starting reconcile operation <7> 2023-08-25 13:07:01.401 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to List modules <7> 2023-08-25 13:07:01.401 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:07:01.401 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:07:01.401 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules?api-version=2022-08-03 <7> 2023-08-25 13:07:01.401 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Getting edge agent config... <7> 2023-08-25 13:07:01.412 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK <7> 2023-08-25 13:07:01.413 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for List modules <6> 2023-08-25 13:07:01.529 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider] - Edge agent attempting to connect to IoT Hub via Amqp_WebSocket_Only... <7> 2023-08-25 13:07:01.530 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to Getting System Info <7> 2023-08-25 13:07:01.530 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:07:01.530 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:07:01.530 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/systeminfo?api-version=2022-08-03 <7> 2023-08-25 13:07:01.543 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK <7> 2023-08-25 13:07:01.544 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for Getting System Info <7> 2023-08-25 13:07:01.544 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Connection status changed to Disconnected_Retrying with reason Communication_Error <4> 2023-08-25 13:07:06.404 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.BackupConfigSource] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead <7> 2023-08-25 13:07:06.405 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Finished reconcile operation <7> 2023-08-25 13:07:11.407 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Starting reconcile operation <7> 2023-08-25 13:07:11.407 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to List modules <7> 2023-08-25 13:07:11.408 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:07:11.408 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:07:11.408 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules?api-version=2022-08-03 <7> 2023-08-25 13:07:11.408 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Getting edge agent config... <7> 2023-08-25 13:07:11.418 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK <7> 2023-08-25 13:07:11.418 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for List modules <4> 2023-08-25 13:07:16.408 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.BackupConfigSource] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead <7> 2023-08-25 13:07:16.408 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Finished reconcile operation <7> 2023-08-25 13:07:21.409 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Starting reconcile operation <7> 2023-08-25 13:07:21.409 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to List modules <7> 2023-08-25 13:07:21.409 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:07:21.410 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:07:21.410 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules?api-version=2022-08-03 <7> 2023-08-25 13:07:21.410 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Getting edge agent config... <7> 2023-08-25 13:07:21.421 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK <7> 2023-08-25 13:07:21.421 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for List modules <4> 2023-08-25 13:07:26.412 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.BackupConfigSource] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead <7> 2023-08-25 13:07:26.412 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Finished reconcile operation <7> 2023-08-25 13:07:31.414 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Starting reconcile operation <7> 2023-08-25 13:07:31.414 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to List modules <7> 2023-08-25 13:07:31.414 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:07:31.414 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:07:31.415 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules?api-version=2022-08-03 <7> 2023-08-25 13:07:31.415 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Getting edge agent config... <7> 2023-08-25 13:07:31.422 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK <7> 2023-08-25 13:07:31.423 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for List modules <7> 2023-08-25 13:07:34.276 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Starting periodic operation Get system resources... <7> 2023-08-25 13:07:34.276 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to Getting System Resources <7> 2023-08-25 13:07:34.276 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:07:34.276 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:07:34.276 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/systeminfo/resources?api-version=2022-08-03 <7> 2023-08-25 13:07:35.297 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK <7> 2023-08-25 13:07:35.297 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for Getting System Resources <7> 2023-08-25 13:07:35.297 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Service.Program] - Successfully completed periodic operation Get system resources <7> 2023-08-25 13:07:36.332 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Connection status changed to Disconnected with reason Retry_Expired <7> 2023-08-25 13:07:36.332 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider] - Retrying connection to IoT Hub. Current retry count 16. <4> 2023-08-25 13:07:36.415 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.BackupConfigSource] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead <7> 2023-08-25 13:07:36.416 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Finished reconcile operation <7> 2023-08-25 13:07:41.417 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Starting reconcile operation <7> 2023-08-25 13:07:41.417 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to List modules <7> 2023-08-25 13:07:41.417 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:07:41.417 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:07:41.417 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules?api-version=2022-08-03 <7> 2023-08-25 13:07:41.418 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Getting edge agent config... <7> 2023-08-25 13:07:41.427 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK <7> 2023-08-25 13:07:41.427 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for List modules <4> 2023-08-25 13:07:46.419 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.BackupConfigSource] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead <7> 2023-08-25 13:07:46.419 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Finished reconcile operation <7> 2023-08-25 13:07:51.420 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Starting reconcile operation <7> 2023-08-25 13:07:51.420 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to List modules <7> 2023-08-25 13:07:51.420 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:07:51.420 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:07:51.420 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules?api-version=2022-08-03 <7> 2023-08-25 13:07:51.420 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Getting edge agent config... <7> 2023-08-25 13:07:51.437 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK <7> 2023-08-25 13:07:51.437 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for List modules <4> 2023-08-25 13:07:56.421 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.BackupConfigSource] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead <7> 2023-08-25 13:07:56.421 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Finished reconcile operation <7> 2023-08-25 13:08:01.421 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Starting reconcile operation <7> 2023-08-25 13:08:01.421 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to List modules <7> 2023-08-25 13:08:01.422 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:08:01.422 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:08:01.422 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules?api-version=2022-08-03 <7> 2023-08-25 13:08:01.422 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Getting edge agent config... <7> 2023-08-25 13:08:01.432 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK <7> 2023-08-25 13:08:01.432 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for List modules <4> 2023-08-25 13:08:06.421 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.BackupConfigSource] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead <7> 2023-08-25 13:08:06.421 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Finished reconcile operation <7> 2023-08-25 13:08:11.421 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Starting reconcile operation <7> 2023-08-25 13:08:11.421 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to List modules <7> 2023-08-25 13:08:11.421 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:08:11.422 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:08:11.422 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules?api-version=2022-08-03 <7> 2023-08-25 13:08:11.422 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Getting edge agent config... <7> 2023-08-25 13:08:11.432 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK <7> 2023-08-25 13:08:11.432 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for List modules <4> 2023-08-25 13:08:16.424 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.BackupConfigSource] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead <7> 2023-08-25 13:08:16.424 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Finished reconcile operation <7> 2023-08-25 13:08:21.424 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Starting reconcile operation <7> 2023-08-25 13:08:21.424 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to List modules <7> 2023-08-25 13:08:21.424 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:08:21.424 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:08:21.424 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules?api-version=2022-08-03 <7> 2023-08-25 13:08:21.424 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Getting edge agent config... <7> 2023-08-25 13:08:21.435 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK <7> 2023-08-25 13:08:21.436 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for List modules <4> 2023-08-25 13:08:26.425 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.BackupConfigSource] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead <7> 2023-08-25 13:08:26.425 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Finished reconcile operation <7> 2023-08-25 13:08:31.426 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Starting reconcile operation <7> 2023-08-25 13:08:31.426 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to List modules <7> 2023-08-25 13:08:31.426 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:08:31.426 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:08:31.426 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules?api-version=2022-08-03 <7> 2023-08-25 13:08:31.426 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Getting edge agent config... <7> 2023-08-25 13:08:31.436 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK <7> 2023-08-25 13:08:31.437 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for List modules <6> 2023-08-25 13:08:36.333 +00:00 [INF] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.ModuleClientProvider] - Edge agent attempting to connect to IoT Hub via Amqp_WebSocket_Only... <7> 2023-08-25 13:08:36.334 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to Getting System Info <7> 2023-08-25 13:08:36.334 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:08:36.334 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:08:36.334 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/systeminfo?api-version=2022-08-03 <7> 2023-08-25 13:08:36.347 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK <7> 2023-08-25 13:08:36.347 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for Getting System Info <7> 2023-08-25 13:08:36.348 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.IoTHub.EdgeAgentConnection] - Connection status changed to Disconnected_Retrying with reason Communication_Error <4> 2023-08-25 13:08:36.425 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.BackupConfigSource] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead <7> 2023-08-25 13:08:36.425 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Finished reconcile operation <7> 2023-08-25 13:08:41.428 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Starting reconcile operation <7> 2023-08-25 13:08:41.428 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Making a Http call to unix:///var/run/iotedge/mgmt.sock to List modules <7> 2023-08-25 13:08:41.428 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connecting socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:08:41.428 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Connected socket /var/run/iotedge/mgmt.sock <7> 2023-08-25 13:08:41.428 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Sending request http://mgmt.sock/modules?api-version=2022-08-03 <7> 2023-08-25 13:08:41.428 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Core.Agent] - Getting edge agent config... <7> 2023-08-25 13:08:41.443 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Util.Uds.HttpUdsMessageHandler] - Response received OK <7> 2023-08-25 13:08:41.443 +00:00 [DBG] [Microsoft.Azure.Devices.Edge.Agent.Edgelet.ModuleManagementHttpClient] - Received a valid Http response from unix:///var/run/iotedge/mgmt.sock for List modules <4> 2023-08-25 13:08:46.431 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Agent.Core.ConfigSources.BackupConfigSource] - Empty edge agent config was received. Attempting to read config from backup (/tmp/edgeAgent/backup.json) instead ```

Additional Information

Please provide any additional information that may be helpful in understanding the issue.

jlian commented 1 year ago

HTTPS should be supported. Are you using something like ZScaler? Did you put the CA cert in the trust bundle config?

@huguesBouvier (on-call) can you help take a look?

huguesBouvier commented 1 year ago

HTTPS should be supported. Do you see anything relevant in the logs at the proxy level?

You could try validating your proxy first: With a command like this for example: openssl s_client -proxy -connect -trace

frankvdbh commented 1 year ago

We are not using ZScaler but a self-made proxy based on Squid. The openssl s_client command does not support proxy addresses with HTTPS... for example openssl s_client -proxy 1.2.3.4:443 -connect iothub-hostname.azure-devices-net:443 -debug this is not supported because it expects the proxy to be HTTP.

I did try with curl commands and this way I can successfully reach external sites, e.g. curl -x "https://proxy" "http://httpbin.org/ip"

I will check for additional debug level logs on the proxy.

frankvdbh commented 1 year ago

Some updates: We configured the proxy now with a full publicly trusted cert, on SSL Labs it is showing as score A with required intermediate certs etc being sent. On the IOT Edge client, there are no trust issues reported when we test the proxy. For the proxy URL we have configured everywhere https://proxy.contoso.com:443

Can we somehow further debug the response received by EdgeAgent to see whether it contains some indication about possible error?

frankvdbh commented 1 year ago

update: Changing upstreamProtocol from AmqpWs to Mqtt or Amqp seems to make the communication working. Looks like it is related to the Websockets handling

github-actions[bot] commented 1 year ago

This issue is being marked as stale because it has been open for 30 days with no activity.

huguesBouvier commented 1 year ago

I am closing the issue, let us know if you need more help on it.