[BUG] Azure.Storage.Blobs cannot connect to azure blob storage on AzureIoTEdge when using Azure IoTEdge API proxy and HTTPS

[sitaalbeMarp]

Library name and version

Azure.Storage.Blobs v>= 12.15.1

Describe the bug

Dear Azure team, currently we are using Azure IoT Edge v.1.4 on Ubuntu Server 20.04 with Azure Blob Storage container service (v.1.4.4-linux-amd64).

We have IoT devices that connects to Azure IoT Edge and also uses Azure Blob Storage on IoT Edge for uploading files.

When we use HTTP protocol with connections strings like: "DefaultEndpointsProtocol=http;BlobEndpoint=http://10.52.118.2:11002/edgeblob;AccountName=edgeblob;AccountKey=<64 bit key>;" everything is working fine.

Also connections from inside other containers with connection string like: "DefaultEndpointsProtocol=http;BlobEndpoint=http://azureblobstorageoniotedge:11002/edgeblob;AccountName=edgeblob;AccountKey=<64 bit key>;" is working fine.

Since we want to use secure protocol like HTTPS, we decided to use Azure IoT Edge API Proxy module: https://learn.microsoft.com/en-us/azure/iot-edge/how-to-configure-api-proxy-module?view=iotedge-1.4

We have successfully configured the latest version available of the proxy and configured the environment variable of the proxy: _BLOB_UPLOAD_ROUTEADDRESS with the value azureblobstorageoniotedge:11002 since the communication between the proxy and the blob container is limited inside docker on the edge device.

We used this connection string for TLS connection on the IoT devices: DefaultEndpointsProtocol=https;BlobEndpoint=https://iotedge/edgeblob;AccountName=edgeblob;AccountKey=<64-bit-key>;"

The endpoint is correctly resolved since the DNS is fine and also certificate are correctly configured on Edge device (EdgeCA is ok). Also the proxy is configured fine and at TLS level we have no errors.

The problem we encountered is this one form the APIs related to interaction with blob storage: RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. Connection: [keep-alive] Content-Length: [0] Date: [Fri, 22 Mar 2024 08:15:35 GMT] Server: [nginx/1.22.1] X-Ms-Error-Code: [AuthenticationFailed] X-Ms-Request-Id: [459adac6-fbb8-43a4-8cbf-93671a39023b] I've noticed that when using https connection string, the SDK does not compute well the signature for authorization. We checked the time and machines are correctly synced. We are using volumes mount for Blob storage persistance on the edge so no errors related on file system permissions are present.

The problem I think it is related to the misunderstanding of the account name:

The problem is that the SDK, when parsing https connection strings, accepts only endpoint in the form https://blob-account-name.blob.ms-cloud-sub-domain. Delving into the SDK code, it seems that the parsing function does not take into account https urls in different forms.

In our case we have a blob endpoint like: https://iotedge/edgeblob, where iotedge is the FQDN name of the edge machine and the edgeblob is the container name.

When using http connection string with IP address it is fine:

Since in this case the parsing in the SDK takes into account the fact that an IP address is explicitly present.

I've tried to update to Azure.Storage.Blobs 12.19.0, but I'm getting a different error related to x-ms-version not being compatible with the blob storage on edge device.

How can I solve both problems?

Expected behavior

When using connection string behing Azure IoTEdge API Proxy from IoT Device DefaultEndpointsProtocol=https;BlobEndpoint=https://iotedge/edgeblob;AccountName=edgeblob;AccountKey=<64-bit-key>;"

The connection from Azure.Storage.Blobs v >= 12.15.1 is fine and the IoT Application can upload and download blobs passing through the proxy.

Actual behavior

When using connection string behing Azure IoTEdge API Proxy from IoT Device DefaultEndpointsProtocol=https;BlobEndpoint=https://iotedge/edgeblob;AccountName=edgeblob;AccountKey=<64-bit-key>;"

The connection from Azure.Storage.Blobs v >= 12.15.1 we face 2 errors:

With Azure.Storage.Blobs v 12.15.1 we have the error: RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. Connection: [keep-alive] Content-Length: [0] Date: [Fri, 22 Mar 2024 08:15:35 GMT] Server: [nginx/1.22.1] X-Ms-Error-Code: [AuthenticationFailed] X-Ms-Request-Id: [459adac6-fbb8-43a4-8cbf-93671a39023b] I've noticed that when using https connection string, the SDK does not compute well the signature for authorization. We checked the time and machines are correctly synced. We are using volumes mount for Blob storage persistance on the edge so no errors related on file system permissions are present.

The problem I think it is related to the misunderstanding of the account name:

The problem is that the SDK, when parsing https connection strings, accepts only endpoint in the form https://blob-account-name.blob.ms-cloud-sub-domain. Delving into the SDK code, it seems that the parsing function does not take into account https urls in different forms.

In our case we have a blob endpoint like: https://iotedge/edgeblob, where iotedge is the FQDN name of the edge machine and the edgeblob is the container name.

When using http connection string with IP address it is fine:

Since in this case the parsing in the SDK takes into account the fact that an IP address is explicitly present.

With Azure.Storage.Blobs v >= 12.16.0 we have the error:

x-ms-version not supported. We suppose that with version 12.16.0 and higer the blob storage on iotedge is not supported with C# sdk

Reproduction Steps

We have successfully configured the latest version available of the proxy and configured the environment variable of the proxy: _BLOB_UPLOAD_ROUTEADDRESS with the value azureblobstorageoniotedge:11002 since the communication between the proxy and the blob container is limited inside docker on the edge device.

We used this connection string for TLS connection on the IoT devices: DefaultEndpointsProtocol=https;BlobEndpoint=https://iotedge/edgeblob;AccountName=edgeblob;AccountKey=<64-bit-key>;" we face RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. Connection: [keep-alive] Content-Length: [0] Date: [Fri, 22 Mar 2024 08:15:35 GMT] Server: [nginx/1.22.1] X-Ms-Error-Code: [AuthenticationFailed] X-Ms-Request-Id: [459adac6-fbb8-43a4-8cbf-93671a39023b] Clocks are ok.

Environment

dotnet 8 applications Azure.Storage.Blobs v >= 12.15.1

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[sitaalbeMarp]

Dear Azure Team, any update on this topic?

Thank you!

[sitaalbeMarp]

Update: Using Azure.Storage.Blob v 12.8.4 and overriding class BlobContainerClient like shown in the below picture: The connection is fine, but the library version is vulnerable so we cannot use it.

[sitaalbeMarp]

Hello! Is there any news on this topic? Thanks and kind regards.