Azure / azure-sdk-for-go

This repository is for active development of the Azure SDK for Go. For consumers of the SDK we recommend visiting our public developer docs at:
https://docs.microsoft.com/azure/developer/go/
MIT License
1.59k stars 821 forks source link

[Storage][pageblob] Page blob implementation is broken, does not set required headers #22474

Closed berndverst closed 6 months ago

berndverst commented 6 months ago

Bug Report

The pageblob module (submodule of azblob) does not set required headers when attempting to upload a page blob to PREMIUM storage

The proxy test which replays existing recordings works fine (run via go test -v --count=1) but running a test against real Azure resources does not work.

I set the Storage Account environment variable and made sure I have Azure Storage Data Owner permission. Authentication happened successfully via the DefaultAzureCredential which used my AzureCLICredential.

https://github.com/Azure/azure-sdk-for-go/blob/476924417d9088bf5779517af24f8375cb567f4a/sdk/storage/azblob/pageblob/examples_test.go#L31-L112

SDK Versions:

        github.com/Azure/azure-sdk-for-go/sdk/azcore v1.9.2
        github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.5.1
        github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.3.1

I compiled the sample (by renaming the function to main and changing the package to main, and changing the container name to something I already had) to a binary m and then ran it like so:

AZURE_STORAGE_ACCOUNT_NAME=cnagstorage AZURE_SDK_GO_LOGGING=all ./m

Here are the full logs with Azure Go SDK debug logs enabled:

[Feb 28 12:46:05.428605] Authentication: Managed Identity Credential will use IMDS managed identity
[Feb 28 12:46:05.429259] Authentication: NewDefaultAzureCredential failed to initialize some credentials:
    EnvironmentCredential: missing environment variable AZURE_TENANT_ID
    WorkloadIdentityCredential: no client ID specified. Check pod configuration or set ClientID in the options
[Feb 28 12:46:05.429316] Retry: =====> Try=1
[Feb 28 12:46:05.429490] Retry: =====> Try=1
[Feb 28 12:46:05.429527] Request: ==> OUTGOING REQUEST (Try=1)
   GET http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=REDACTED
   Metadata: REDACTED
   User-Agent: azsdk-go-azidentity/v1.5.1 (go1.22.0; darwin)

[Feb 28 12:46:05.429988] Response: ==> REQUEST/RESPONSE (Try=1/314.125µs, OpTime=342.625µs) -- REQUEST ERROR
   GET http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=REDACTED
   Metadata: REDACTED
   User-Agent: azsdk-go-azidentity/v1.5.1 (go1.22.0; darwin)
   --------------------------------------------------------------------------------
   ERROR:
Get "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fstorage.azure.com": dial tcp 169.254.169.254:80: connect: host is down
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*logPolicy).Do()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/runtime/policy_logging.go:140
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/internal/exported/request.go:107
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*httpTracePolicy).Do()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/runtime/policy_http_trace.go:93
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/internal/exported/request.go:107
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.httpHeaderPolicy()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/runtime/policy_http_header.go:31
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.PolicyFunc.Do()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/internal/exported/request.go:177
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/internal/exported/request.go:107
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*retryPolicy).Do()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/runtime/policy_retry.go:126
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/internal/exported/request.go:107
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.telemetryPolicy.Do()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/runtime/policy_telemetry.go:70
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/internal/exported/request.go:107
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.includeResponsePolicy()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/runtime/policy_include_response.go:19
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.PolicyFunc.Do()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/internal/exported/request.go:177
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/internal/exported/request.go:107
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.Pipeline.Do()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/internal/exported/pipeline.go:76
github.com/Azure/azure-sdk-for-go/sdk/azidentity.(*managedIdentityClient).authenticate()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azidentity@v1.5.1/managed_identity_client.go:188
github.com/Azure/azure-sdk-for-go/sdk/azidentity.(*managedIdentityClient).provideToken()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azidentity@v1.5.1/managed_identity_client.go:173
github.com/AzureAD/microsoft-authentication-library-for-go/apps/internal/oauth.(*Client).Credential()
    /Users/bverst/go/pkg/mod/github.com/!azure!a!d/microsoft-authentication-library-for-go@v1.2.1/apps/internal/oauth/oauth.go:113
github.com/AzureAD/microsoft-authentication-library-for-go/apps/confidential.Client.AcquireTokenByCredential()
    /Users/bverst/go/pkg/mod/github.com/!azure!a!d/microsoft-authentication-library-for-go@v1.2.1/apps/confidential/confidential.go:675
github.com/Azure/azure-sdk-for-go/sdk/azidentity.(*confidentialClient).GetToken()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azidentity@v1.5.1/confidential_client.go:106
github.com/Azure/azure-sdk-for-go/sdk/azidentity.(*ManagedIdentityCredential).GetToken()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azidentity@v1.5.1/managed_identity_credential.go:117
github.com/Azure/azure-sdk-for-go/sdk/azidentity.(*timeoutWrapper).GetToken()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azidentity@v1.5.1/default_azure_credential.go:185
github.com/Azure/azure-sdk-for-go/sdk/azidentity.(*ChainedTokenCredential).GetToken()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azidentity@v1.5.1/chained_token_credential.go:92
github.com/Azure/azure-sdk-for-go/sdk/azidentity.(*DefaultAzureCredential).GetToken()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azidentity@v1.5.1/default_azure_credential.go:146
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.acquire()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/runtime/policy_bearer_token.go:38
github.com/Azure/azure-sdk-for-go/sdk/internal/temporal.(*Resource[...]).Get()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/internal@v1.5.2/temporal/resource.go:95
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*BearerTokenPolicy).Do.(*BearerTokenPolicy).authenticateAndAuthorize.func1()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/runtime/policy_bearer_token.go:65
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared.(*storageAuthorizer).onRequest()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v1.3.1/internal/shared/challenge_policy.go:34
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*BearerTokenPolicy).Do()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/runtime/policy_bearer_token.go:89
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/internal/exported/request.go:107
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*retryPolicy).Do()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/runtime/policy_retry.go:121
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next()
    /Users/bverst/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.9.2/internal/exported/request.go:107

[Feb 28 12:46:05.438546] Retry: error Get "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fstorage.azure.com": dial tcp 169.254.169.254:80: connect: host is down
[Feb 28 12:46:05.438579] Retry: End Try #1, Delay=1.963753246s
[Feb 28 12:46:06.430436] Retry: abort due to context deadline exceeded
[Feb 28 12:46:07.145885] Authentication: AzureCLICredential.GetToken() acquired a token for scope "https://storage.azure.com/.default"
[Feb 28 12:46:07.145907] Authentication: DefaultAzureCredential authenticated with AzureCLICredential
[Feb 28 12:46:07.145937] Request: ==> OUTGOING REQUEST (Try=1)
   PUT https://cnagstorage.blob.core.windows.net/xdsir/test_page_blob.vhd
   Accept: application/xml
   Authorization: REDACTED
   Content-Length: 0
   User-Agent: azsdk-go-azblob/v1.3.1 (go1.22.0; darwin)
   x-ms-blob-content-length: REDACTED
   x-ms-blob-type: REDACTED
   x-ms-version: REDACTED

[Feb 28 12:46:07.324525] Response: ==> REQUEST/RESPONSE (Try=1/178.557916ms, OpTime=178.576666ms) -- RESPONSE RECEIVED
   PUT https://cnagstorage.blob.core.windows.net/xdsir/test_page_blob.vhd
   Accept: application/xml
   Authorization: REDACTED
   Content-Length: 0
   User-Agent: azsdk-go-azblob/v1.3.1 (go1.22.0; darwin)
   x-ms-blob-content-length: REDACTED
   x-ms-blob-type: REDACTED
   x-ms-version: REDACTED
   --------------------------------------------------------------------------------
   RESPONSE Status: 201 Created
   Content-Length: 0
   Date: Wed, 28 Feb 2024 20:46:06 GMT
   Etag: "0x8DC389E49C6D5FF"
   Last-Modified: Wed, 28 Feb 2024 20:46:07 GMT
   Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
   X-Ms-Request-Id: c601187b-401c-0016-3087-6a489c000000
   X-Ms-Request-Server-Encrypted: REDACTED
   X-Ms-Version: REDACTED

[Feb 28 12:46:07.324541] Retry: response 201
[Feb 28 12:46:07.324544] Retry: exit due to non-retriable status code
[Feb 28 12:46:07.324566] Retry: =====> Try=1
[Feb 28 12:46:07.324582] Request: ==> OUTGOING REQUEST (Try=1)
   PUT https://cnagstorage.blob.core.windows.net/xdsir/test_page_blob.vhd?comp=REDACTED
   Accept: application/xml
   Authorization: REDACTED
   Content-Length: 512
   Content-Type: application/octet-stream
   User-Agent: azsdk-go-azblob/v1.3.1 (go1.22.0; darwin)
   x-ms-page-write: REDACTED
   x-ms-version: REDACTED

[Feb 28 12:46:07.362055] Response: ==> REQUEST/RESPONSE (Try=1/37.455583ms, OpTime=37.4665ms) -- RESPONSE RECEIVED
   PUT https://cnagstorage.blob.core.windows.net/xdsir/test_page_blob.vhd?comp=REDACTED
   Accept: application/xml
   Authorization: REDACTED
   Content-Length: 512
   Content-Type: application/octet-stream
   User-Agent: azsdk-go-azblob/v1.3.1 (go1.22.0; darwin)
   x-ms-page-write: REDACTED
   x-ms-version: REDACTED
   --------------------------------------------------------------------------------
   RESPONSE Status: 400 An HTTP header that's mandatory for this request is not specified.
   Content-Length: 292
   Content-Type: application/xml
   Date: Wed, 28 Feb 2024 20:46:06 GMT
   Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
   X-Ms-Error-Code: REDACTED
   X-Ms-Request-Id: c601187c-401c-0016-3187-6a489c000000
   X-Ms-Version: REDACTED

[Feb 28 12:46:07.362065] Retry: response 400
[Feb 28 12:46:07.362067] Retry: exit due to non-retriable status code
2024/02/28 12:46:07 PUT https://cnagstorage.blob.core.windows.net/xdsir/test_page_blob.vhd
--------------------------------------------------------------------------------
RESPONSE 400: 400 An HTTP header that's mandatory for this request is not specified.
ERROR CODE: MissingRequiredHeader
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?>
<Error><Code>MissingRequiredHeader</Code><Message>An HTTP header that&apos;s mandatory for this request is not specified.
RequestId:c601187c-401c-0016-3187-6a489c000000
Time:2024-02-28T20:46:07.3886562Z</Message><HeaderName>Range</HeaderName></Error>
--------------------------------------------------------------------------------
berndverst commented 6 months ago

@tanyasethi-msft could you please take a look at this? Feel free to ping me internally.

In case the type of Storage account plays a role, here are the details:

 az storage account show -n cnagstorage
{
  "accessTier": "Hot",
  "accountMigrationInProgress": null,
  "allowBlobPublicAccess": false,
  "allowCrossTenantReplication": false,
  "allowSharedKeyAccess": null,
  "allowedCopyScope": null,
  "azureFilesIdentityBasedAuthentication": null,
  "blobRestoreStatus": null,
  "creationTime": "2024-02-28T01:58:14.017250+00:00",
  "customDomain": null,
  "defaultToOAuthAuthentication": null,
  "dnsEndpointType": null,
  "enableHttpsTrafficOnly": true,
  "enableNfsV3": null,
  "encryption": {
    "encryptionIdentity": null,
    "keySource": "Microsoft.Storage",
    "keyVaultProperties": null,
    "requireInfrastructureEncryption": null,
    "services": {
      "blob": {
        "enabled": true,
        "keyType": "Account",
        "lastEnabledTime": "2024-02-28T01:58:14.235960+00:00"
      },
      "file": {
        "enabled": true,
        "keyType": "Account",
        "lastEnabledTime": "2024-02-28T01:58:14.235960+00:00"
      },
      "queue": null,
      "table": null
    }
  },
  "extendedLocation": null,
  "failoverInProgress": null,
  "geoReplicationStats": null,
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXX/resourceGroups/beverst-dev/providers/Microsoft.Storage/storageAccounts/cnagstorage",
  "identity": null,
  "immutableStorageWithVersioning": null,
  "isHnsEnabled": null,
  "isLocalUserEnabled": null,
  "isSftpEnabled": null,
  "isSkuConversionBlocked": null,
  "keyCreationTime": {
    "key1": "2024-02-28T01:58:14.095336+00:00",
    "key2": "2024-02-28T01:58:14.095336+00:00"
  },
  "keyPolicy": null,
  "kind": "StorageV2",
  "largeFileSharesState": null,
  "lastGeoFailoverTime": null,
  "location": "westus3",
  "minimumTlsVersion": "TLS1_0",
  "name": "cnagstorage",
  "networkRuleSet": {
    "bypass": "AzureServices",
    "defaultAction": "Allow",
    "ipRules": [],
    "ipv6Rules": [],
    "resourceAccessRules": null,
    "virtualNetworkRules": []
  },
  "primaryEndpoints": {
    "blob": "https://cnagstorage.blob.core.windows.net/",
    "dfs": null,
    "file": null,
    "internetEndpoints": null,
    "microsoftEndpoints": null,
    "queue": null,
    "table": null,
    "web": "https://cnagstorage.z1.web.core.windows.net/"
  },
  "primaryLocation": "westus3",
  "privateEndpointConnections": [],
  "provisioningState": "Succeeded",
  "publicNetworkAccess": null,
  "resourceGroup": "beverst-dev",
  "routingPreference": null,
  "sasPolicy": null,
  "secondaryEndpoints": null,
  "secondaryLocation": null,
  "sku": {
    "name": "Premium_ZRS",
    "tier": "Premium"
  },
  "statusOfPrimary": "available",
  "statusOfSecondary": null,
  "storageAccountSkuConversionStatus": null,
  "tags": {},
  "type": "Microsoft.Storage/storageAccounts"
}
tanyasethi-msft commented 6 months ago

Hi @berndverst , Thanks for reporting this issue! This seems more like a documentation bug here, where offset: 0, count: 0 should not be a valid example. This code piece here checks if both offset and count are zero, and if yes, it considers no range, and does not add it to the header list.

I will make changes in the documentation.

berndverst commented 6 months ago

@tanyasethi-msft could you please provide a complete example that reads an arbitrary file or buffer and uploads the entire thing?

That's really the sample everyone is missing.

FWIW, I actually wrote other code to change offset and count values.. and everytime it would complain about other missing headers or incorrect header values.

A complete example is truly needed.

For example some unknowns here: Does a file have to be uploaded as multiple pages? If so what does the page size need to be?

We really need some clearer samples, and ideally there would be some convenience methods added to the SDK that can upload an entire file or buffer as pages using some sane defaults.