azblob fails to list containers or fetch container's content on AzureStack

r4f4 commented 1 year ago

Bug Report

import path of package in question: github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
SDK version:
```
$: cat go.mod 
module serial
```

go 1.19

require ( github.com/Azure/azure-sdk-for-go/sdk/azcore v1.4.0 github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.1 github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.2.0 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 github.com/Azure/go-autorest/autorest v0.11.28 )

require ( github.com/Azure/azure-sdk-for-go/sdk/internal v1.1.2 // indirect github.com/Azure/go-autorest v14.2.0+incompatible // indirect github.com/Azure/go-autorest/autorest/adal v0.9.20 // indirect github.com/Azure/go-autorest/autorest/date v0.3.0 // indirect github.com/Azure/go-autorest/logger v0.2.1 // indirect github.com/Azure/go-autorest/tracing v0.6.0 // indirect github.com/AzureAD/microsoft-authentication-library-for-go v0.8.1 // indirect github.com/golang-jwt/jwt/v4 v4.4.2 // indirect github.com/google/uuid v1.1.1 // indirect github.com/kylelemons/godebug v1.1.0 // indirect github.com/pkg/browser v0.0.0-20210115035449-ce105d075bb4 // indirect golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa // indirect golang.org/x/net v0.7.0 // indirect golang.org/x/sys v0.5.0 // indirect golang.org/x/text v0.7.0 // indirect )

- output of `go version`:
```bash
$: go version
go version go1.19.6 linux/amd64

What happened? Trying to fetch containers list or the contents of a container with azblob fail on AzureStack environments (I tried WWT and ASH). The error is

faild to list containers: GET https://seriallogaccnt.blob.<endpoint suffix redacted>/
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidHeaderValue
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidHeaderValue</Code><Message>The value for one of the HTTP headers is not in the correct format.
RequestId:2894f877-2587-0235-76b7-1adf179cf800
Time:2023-03-22T16:31:56.7529289Z</Message><HeaderName>x-ms-version</HeaderName><HeaderValue>2020-10-02</HeaderValue></Error>
--------------------------------------------------------------------------------

What did you expect or want to happen? The operation is successful as when using the az cli:

$: az storage container list --account-name seriallogaccnt --account-key "<REDACTED>"
[
{
"metadata": null,
"name": "seriallogcontainer",
"properties": {
  "etag": "\"0x8DB2AEC2FD1B806\"",
  "hasImmutabilityPolicy": "false",
  "hasLegalHold": "false",
  "lastModified": "2023-03-22T15:43:30+00:00",
  "lease": {
    "duration": null,
    "state": null,
    "status": null
  },
  "leaseDuration": null,
  "leaseState": "available",
  "leaseStatus": "unlocked",
  "publicAccess": null
}
}
]

How can we reproduce it? With the following code:
```
package main
```

import ( "context" "fmt" "time"

"github.com/Azure/azure-sdk-for-go/sdk/azcore"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/arm"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/cloud"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/policy"
"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
"github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob"
azureenv "github.com/Azure/go-autorest/autorest/azure"

)

const ( armEndpoint = "https://management." subscriptionID = "" clientID = "" clientSecret = "" tenantID = "" resourceGroup = "test-serial-log" storageAccountName = "seriallogaccnt" accountURL = "https://seriallogaccnt.blob./" containerName = "seriallogcontainer" )

func main() { cloudEnv, err := azureenv.EnvironmentFromURL(armEndpoint) if err != nil { fmt.Printf("failed to load environment from endpoint: %v\n", err) return }

cloudConfig := cloud.Configuration{
    ActiveDirectoryAuthorityHost: cloudEnv.ActiveDirectoryEndpoint,
    Services: map[cloud.ServiceName]cloud.ServiceConfiguration{
        cloud.ResourceManager: {
            Audience: cloudEnv.TokenAudience,
            Endpoint: cloudEnv.ResourceManagerEndpoint,
        },
    },
}

options := azidentity.ClientSecretCredentialOptions{
    ClientOptions: azcore.ClientOptions{
        Cloud: cloudConfig,
    },
}

cred, err := azidentity.NewClientSecretCredential(tenantID, clientID, clientSecret, &options)
if err != nil {
    fmt.Printf("failed to get client secret credential: %v\n", err)
    return
}

accntClientOptions := arm.ClientOptions{
    ClientOptions: policy.ClientOptions{
        APIVersion: "2019-06-01",
        Cloud:      cloudConfig,
    },
}
accntClient, err := armstorage.NewAccountsClient(subscriptionID, cred, &accntClientOptions)
if err != nil {
    fmt.Printf("failed to get accounts client: %v\n", err)
    return
}

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
defer cancel()

keyCreds, err := getSharedKeyCredentials(ctx, accntClient)
if err != nil {
    fmt.Printf("failed to get sharedKey credentials: %v\n", err)
    return
}

blobClient, err := azblob.NewClientWithSharedKeyCredential(accountURL, keyCreds[0], nil)
if err != nil {
    fmt.Printf("failed to get blob client: %v\n", err)
    return
}

cPager := blobClient.NewListContainersPager(nil)
for cPager.More() {
    page, err := cPager.NextPage(ctx)
    if err != nil {
        fmt.Printf("faild to list containers: %v\n", err)
        return
    }
    fmt.Printf("Found container %v\n", page)
}
pager := blobClient.NewListBlobsFlatPager(containerName, nil)
for pager.More() {
    page, err := pager.NextPage(ctx)
    if err != nil {
        fmt.Printf("failed getting blob page: %v\n", err)
        return
    }
    fmt.Printf("Found blob %v\n", page)
}

}

func getSharedKeyCredentials(ctx context.Context, accntClient armstorage.AccountsClient) ([]azblob.SharedKeyCredential, error) { var creds []*azblob.SharedKeyCredential res, err := accntClient.ListKeys(ctx, resourceGroup, storageAccountName, nil) if err != nil { return nil, err }

for _, key := range res.Keys {
    if key.Value == nil {
        continue
    }
    sharedKeyCredential, err := azblob.NewSharedKeyCredential(storageAccountName, *key.Value)
    if err != nil {
        fmt.Printf("failed to get shared key: %s\n", err.Error())
        continue
    }
    creds = append(creds, sharedKeyCredential)
}
return creds, nil

}

- Anything we should know about your environment: When setting up cloud in the az cli I did

az cloud update --profile 2019-03-01-hybrid



<!--
Thanks!
-->

ghost commented 1 year ago

Hi @r4f4. Thank you for your feedback and we will look into it soon. Meanwhile, feel free to share your experience using the Azure SDK in this survey.

vibhansa-msft commented 1 year ago

As per the logs backend is returning back error for the service version: <HeaderName>x-ms-version</HeaderName><HeaderValue>2020-10-02</HeaderValue> Can you validate this service version is supported by your AzureStack or it running on an older sv.

r4f4 commented 1 year ago

As per the logs backend is returning back error for the service version: <HeaderName>x-ms-version</HeaderName><HeaderValue>2020-10-02</HeaderValue> Can you validate this service version is supported by your AzureStack or it running on an older sv.

I have filed a support ticket asking for that information. I'll let you know as soon as I have it.

r4f4 commented 1 year ago

@vibhansa-msft but let's say we find out it's running an old SV. Does azblob need to use 2020-10-02 given that I can use the az cli to fetch containers information in our env?

vibhansa-msft commented 1 year ago

If AzureStack is running an older version and AzBlob uses later service version in HTTP headers then its expected that server will reject the request. I am not much aware on what version azcli uses, may be you can check with AzCli team on this.

vibhansa-msft commented 1 year ago

AzBlob is storage SDK for blobs and depending upon what service version we based it on, this value cannot be changed. As current AzBlob code is STG78 compliant it's using exactly the version service expects running on STG78+.

r4f4 commented 1 year ago

AzBlob is storage SDK for blobs and depending upon what service version we based it on, this value cannot be changed. As current AzBlob code is STG78 compliant it's using exactly the version service expects running on STG78+.

That's good to know, thanks. It means we will need to document that certain functionalities might not be available for customers unless they are running AzureStack with a compatible service version >= 2020-10-02.

chlislb commented 1 year ago

From this link https://learn.microsoft.com/en-us/azure-stack/user/azure-stack-acs-differences?view=azs-2206#api-version we can see the latest version is 2019-07-07.

I tried to use ClientOptions API version as below showed, but it failed either. opt := &container.ClientOptions{ ClientOptions: azcore.ClientOptions{ APIVersion: "2019-07-07", Cloud: c, Retry: policy.RetryOptions{ MaxRetries: conf.PipelineConfig.MaxTries, TryTimeout: time.Duration(conf.PipelineConfig.TryTimeout), RetryDelay: time.Duration(conf.PipelineConfig.RetryDelay), MaxRetryDelay: time.Duration(conf.PipelineConfig.MaxRetryDelay), }, Telemetry: policy.TelemetryOptions{ ApplicationID: "Thanos", }, Transport: &http.Client{Transport: dt}, }, }

the reason is that in the container.go, when creating newpipeline by using runtime.pipelineoptions{}. which cause checked fail later. func NewClientWithSharedKeyCredential(containerURL string, cred SharedKeyCredential, options ClientOptions) (*Client, error) { authPolicy := exported.NewSharedKeyCredPolicy(cred) conOptions := shared.GetClientOptions(options) conOptions.PerRetryPolicies = append(conOptions.PerRetryPolicies, authPolicy) pl := runtime.NewPipeline(exported.ModuleName, exported.ModuleVersion, runtime.PipelineOptions{}, &conOptions.ClientOptions)

return (*Client)(base.NewContainerClient(containerURL, pl, cred)), nil

}

vibhansa-msft commented 1 year ago

You can try creating a new per retry policy that sets the service-version header to the older one or one you need and that might do the trick. Just try it out and let us know if that helps. Other than that there is no other way to override the service version that client creates by default.

r4f4 commented 1 year ago

As per the logs backend is returning back error for the service version: <HeaderName>x-ms-version</HeaderName><HeaderValue>2020-10-02</HeaderValue> Can you validate this service version is supported by your AzureStack or it running on an older sv.

I have filed a support ticket asking for that information. I'll let you know as soon as I have it.

@vibhansa-msft It took a while but the admin finally answered: "Current software version is 1.2206.2.52"

siminsavani-msft commented 1 year ago

Hi @r4f4 , that does not look like valid service version. It should look something like this: 2020-10-02.

Please check out the ClientOptions that allows service version override (https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/azcore/policy/policy.go#L31) and try out the solution mentioned by @vibhansa-msft.

You can try creating a new per retry policy that sets the service-version header to the older one or one you need and that might do the trick. Just try it out and let us know if that helps. Other than that there is no other way to override the service version that client creates by default.

vibhansa-msft commented 1 year ago

Closing this as there are no updates. Feel free to reopen if the suggested workaround does not work for you.

Azure / azure-sdk-for-go

azblob fails to list containers or fetch container's content on AzureStack #20456

Bug Report