search

Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.53k stars 2.76k forks source link

Accessing blob container fails with auth error, when using Azure Front Door CDN #29246

Closed StannieV closed 1 year ago

StannieV commented 1 year ago

Describe the bug

In my solution, I want to create a SAS Uri for a blob item that is accessed via an Azure Front Door (AFD) endpoint. The blob container cannot be publicly accessible and therefore the SAS is needed. After creating a SAS for a blob item, you can replace the hostname of the blob URL E.g ‘https://.blob.core.windows.net’ (from the complete SAS) with the hostname of the AFD endpoint. E.g: ‘https:// .z01.azurefd.net’. With this new URL, you can access the blob item (even download it in the browser) through the AFD endpoint and use CDN.

In code, it seemed a clumsy solution first to create a SAS and then replace the hostname of the URL in the code. After some searching, I found a solution in creating a ‘BlobServiceClient’ with a different service URL, which points to the AFD endpoint. At first glance, this seems great. With this, I only have to change the initialization of the blob object(s) in my startup logic and keep my logic in the application the same. Unfortunately, this is not always working. In half of the cases, it works and in the other half, an exception is thrown in the code.

Why is this exception thrown and should it work as I intend? Or, are there better alternatives for what I want to achieve?

The exception that is produced is:

Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:abba20e0-f01e-004d-1969-52f96d000000
Time:2023-03-09T09:26:42.6689043Z
Status: 403 (Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.)
ErrorCode: AuthenticationFailed

Additional Information:
AuthenticationErrorDetail: The MAC signature found in the HTTP request 'C7MEyGoAuJVMoVG7whS58c46CW/2JUlPEiQoylCt4ok=' is not the same as any computed signature. Server used following string to sign: 'GET

bytes=0-8388607
x-ms-client-request-id:9798c48e-0480-4e8e-990e-2a5f615ec103
x-ms-date:Thu, 09 Mar 2023 09:26:42 GMT
x-ms-return-client-request-id:true
x-ms-version:2021-12-02
/<NAME>/machineimages
comp:list
restype:container'.

Content:
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:abba20e0-f01e-004d-1969-52f96d000000
Time:2023-03-09T09:26:42.6689043Z</Message><AuthenticationErrorDetail>The MAC signature found in the HTTP request 'C7MEyGoAuJVMoVG7whS58c46CW/2JUlPEiQoylCt4ok=' is not the same as any computed signature. Server used following string to sign: 'GET

bytes=0-8388607
x-ms-client-request-id:9798c48e-0480-4e8e-990e-2a5f615ec103
x-ms-date:Thu, 09 Mar 2023 09:26:42 GMT
x-ms-return-client-request-id:true
x-ms-version:2021-12-02
/<NAME>/machineimages
comp:list
restype:container'.</AuthenticationErrorDetail></Error>

Headers:
Date: Thu, 09 Mar 2023 09:26:42 GMT
Connection: keep-alive
x-ms-request-id: abba20e0-f01e-004d-1969-52f96d000000
x-ms-error-code: AuthenticationFailed
Access-Control-Expose-Headers: REDACTED
Access-Control-Allow-Origin: *
x-azure-ref: REDACTED
X-Cache: REDACTED
Content-Type: application/xml
Content-Length: 800

The following C# code reproduces (sometimes) the exception:

using System;
using System.Threading.Tasks;
using Azure.Storage;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;

namespace ConsoleApp23
{
    internal class Program
    {
        static async Task Main(string[] args)
        {
            try
            {
                //Note1: Add a random image to the 'machineimages' container before running the code

                var accountName = "mystoragetst01";
                var accountKey = "*****************************************************************";

                //Note3: This is the original blob url that works without problems
                //string serviceUrl = "https://mystoragetst01.blob.core.windows.net";

                //Note4: This is the Azure Front Door url (Update the after running the BICEP script!) which throw regularly (~50%) an exception when using this.
                string serviceUrl = "https://mystoragetst01-?????????.z01.azurefd.net"; //TODO: Update url based on AFD endpoint after running the BICEP script!

                string containerName = "machineimages";

                StorageSharedKeyCredential credential = new StorageSharedKeyCredential(accountName, accountKey);
                BlobServiceClient blobServiceClient = new BlobServiceClient(new Uri(serviceUrl), credential);
                BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(containerName);

                await foreach (BlobItem blobItem in containerClient.GetBlobsAsync()) //--> This line will produce the exception: 'Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.'
                {
                    Console.WriteLine(blobItem.Name);
                }
            }
            catch (Exception exception)
            {
                Console.WriteLine(exception);
            }
        }
    }
}

Project file:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net7.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Azure.Storage.Blobs" Version="12.15.0" />
  </ItemGroup>

</Project>

The following BICEP script creates the Storage Account and the Azure Front Door (I composed it from multiple files so it isn’t production code):

param location string = resourceGroup().location

var storageAccountName = 'mystoragetst01'

resource storageAccountResource 'Microsoft.Storage/storageAccounts@2022-09-01' = {
  name: storageAccountName
  location: location
  kind: 'StorageV2'
  sku: {
    name: 'Standard_LRS'
  }
  properties: {
    minimumTlsVersion: 'TLS1_2'
    allowBlobPublicAccess: false
    allowSharedKeyAccess: true
    supportsHttpsTrafficOnly: true
    accessTier: 'Hot'
  }
}

//CORS rules
var corsProperty = {
  corsRules:[
    {
      allowedOrigins: [ 
        '*' 
      ]
      allowedHeaders: [
        '*'
      ]
      allowedMethods: [
        'GET'
      ]
      maxAgeInSeconds: 0
      exposedHeaders: [
         '*'
      ]
    }
  ]
}

//Set Data protection/Enable soft delete for blobs
resource blobServicesResource 'Microsoft.Storage/storageAccounts/blobServices@2022-09-01' = {
  parent: storageAccountResource
  name: 'default'
  properties: {
    containerDeleteRetentionPolicy: {
      enabled: true
      days: 14
    }
    deleteRetentionPolicy: {
      enabled: true
      days: 14
    }
    cors: corsProperty
  }
}

//Create container: 
resource containerResource 'Microsoft.Storage/storageAccounts/blobServices/containers@2022-09-01' = {
  name: '${storageAccountName}/default/machineimages'
  dependsOn: [

  ]
}

//Set Lifecycle management
resource storageAccountName_default 'Microsoft.Storage/storageAccounts/managementPolicies@2022-09-01' = {
  name: '${storageAccountName}/default'
  dependsOn: [
    storageAccountResource
  ]
  properties: {
    policy: {
      rules: [
        {
          enabled: true
          name: 'DeleteBlobInMessagesAfter186Days'
          type: 'Lifecycle'
          definition: {
            actions: {
              baseBlob: {
                delete: {
                  daysAfterModificationGreaterThan: 186
                }
              }
            }
            filters: {
              blobTypes: [
                'blockBlob'
              ]
              prefixMatch: [
                'messages'
              ]
            }
          }
        }
      ]
    }
  }
}

//Create Azure Front Door
var frontDoorName = 'MyAFDTest01'

resource azureFrontDoorResource 'Microsoft.Cdn/profiles@2022-11-01-preview' = {
  name: frontDoorName
  location: 'Global'
  sku: {
    name: 'Premium_AzureFrontDoor'
  }
  properties: {
    originResponseTimeoutSeconds: 60
    extendedProperties: {
    }
  }
}

//Retrieves host name by substring 'https://mystoragetst01.blob.core.windows.net/' into 'mystoragetst01.blob.core.windows.net'
var storageAccountHostName = substring(storageAccountResource.properties.primaryEndpoints.blob, 8, length(storageAccountResource.properties.primaryEndpoints.blob) - 8 - 1)

//Add Endpoint to Azure Front Door
var endpointName = storageAccountName
var originGroupName = 'MyStorageTstOriginGroup'
var originName = 'MyStorageTstOrigin'
var originHostName = storageAccountHostName
var endpointRouteName = 'MyStorageTstRoute'
var webApplicationFirewallPolicyName = 'MyWafTest01'
var webApplicationFirewallPolicyMode = 'Detection'
var securityPolicyName = 'MyStorageTstSecurityPolicy'

//AFD Endpoint
resource endpointResource 'Microsoft.Cdn/profiles/afdendpoints@2022-11-01-preview' = {
  parent: azureFrontDoorResource
  name: toLower(endpointName)
  location: 'Global'
  properties: {
    enabledState: 'Enabled'
  }
}

//ADF Origin Group
resource originGroupResource 'Microsoft.Cdn/profiles/origingroups@2022-11-01-preview' = {
  parent: azureFrontDoorResource
  name: originGroupName
  properties: {
    loadBalancingSettings: {
      sampleSize: 4
      successfulSamplesRequired: 3
      additionalLatencyInMilliseconds: 50
    }
    healthProbeSettings: {
      probePath: '/'
      probeRequestType: 'HEAD'
      probeProtocol: 'Http'
      probeIntervalInSeconds: 100
    }
    sessionAffinityState: 'Disabled'
  }
}

//ADF Origin
resource originResource 'Microsoft.Cdn/profiles/origingroups/origins@2022-11-01-preview' = {
  parent: originGroupResource
  name: originName
  properties: {
    hostName: toLower(originHostName)
    httpPort: 80
    httpsPort: 443
    originHostHeader: toLower(originHostName)
    priority: 1
    weight: 1000
    enabledState: 'Enabled'
    enforceCertificateNameCheck: true
  }
}

//ADF Endpoint Route
resource routeResource 'Microsoft.Cdn/profiles/afdendpoints/routes@2022-11-01-preview' = {
  parent: endpointResource
  name: endpointRouteName
  properties: {
    cacheConfiguration: {
      compressionSettings: {
        isCompressionEnabled: true
        contentTypesToCompress: [
          'application/eot'
          'application/font'
          'application/font-sfnt'
          'application/javascript'
          'application/json'
          'application/opentype'
          'application/otf'
          'application/pkcs7-mime'
          'application/truetype'
          'application/ttf'
          'application/vnd.ms-fontobject'
          'application/xhtml+xml'
          'application/xml'
          'application/xml+rss'
          'application/x-font-opentype'
          'application/x-font-truetype'
          'application/x-font-ttf'
          'application/x-httpd-cgi'
          'application/x-javascript'
          'application/x-mpegurl'
          'application/x-opentype'
          'application/x-otf'
          'application/x-perl'
          'application/x-ttf'
          'font/eot'
          'font/ttf'
          'font/otf'
          'font/opentype'
          'image/svg+xml'
          'text/css'
          'text/csv'
          'text/html'
          'text/javascript'
          'text/js'
          'text/plain'
          'text/richtext'
          'text/tab-separated-values'
          'text/xml'
          'text/x-script'
          'text/x-component'
          'text/x-java-source'
        ]
      }
      queryStringCachingBehavior: 'IgnoreQueryString'
    }
    customDomains: []
    originGroup: {
      id: originGroupResource.id
    }
    ruleSets: []
    supportedProtocols: [
      'Https'
    ]
    patternsToMatch: [
      '/*'
    ]
    forwardingProtocol: 'HttpsOnly'
    linkToDefaultDomain: 'Enabled'
    httpsRedirect: 'Disabled'
    enabledState: 'Enabled'
  }
}

//Create Web Application Firewall
resource frontdoorWebApplicationFirewallPolicyResource 'Microsoft.Network/frontdoorwebapplicationfirewallpolicies@2022-05-01' = {
  name: webApplicationFirewallPolicyName
  location: 'Global'
  sku: {
    name: 'Premium_AzureFrontDoor'
  }
  properties: {
    policySettings: {
      enabledState: 'Enabled'
      mode: webApplicationFirewallPolicyMode
      customBlockResponseStatusCode: 403
      requestBodyCheck: 'Enabled'
    }
    customRules: {
      rules: []
    }
    managedRules: {
      managedRuleSets: [
        {
          ruleSetType: 'Microsoft_DefaultRuleSet'
          ruleSetVersion: '2.0'
          ruleSetAction: 'Block'
          ruleGroupOverrides: []
          exclusions: []
        }
        {
          ruleSetType: 'Microsoft_BotManagerRuleSet'
          ruleSetVersion: '1.0'
          ruleGroupOverrides: []
          exclusions: []
        }
      ]
    }
  }
}

//Web Application Firewall connection to the Endpoint
resource securityPolicyResource 'Microsoft.Cdn/profiles/securitypolicies@2022-11-01-preview' = {
  parent: azureFrontDoorResource
  name: securityPolicyName
  properties: {
    parameters: {
      wafPolicy: {
        id: frontdoorWebApplicationFirewallPolicyResource.id
      }
      associations: [
        {
          domains: [
            {
              id: endpointResource.id
            }
          ]
          patternsToMatch: [
            '/*'
          ]
        }
      ]
      type: 'WebApplicationFirewall'
    }
  }
}
kashifkhan commented 1 year ago

Hi @StannieV , just wanted to clarify are you seeing this error when using the Python SDK or the .NET one ? If its the latter then the .net repo is located here

StannieV commented 1 year ago

You are right, it is in the .Net library. I'll close this topic and created a new one in the .net repo.