azure_native.storage.listStorageAccountKeys throws error even with valid credentials

[TomMakes]

What happened?

After updating Pulumi and Azure Native versions to resolve a bug with Microsoft.TimeSeriesInsights (using the solution provided here) I have encountered another bug where listStorageAccountKeys seems to be throwing an error:

error: Running program '*\*\iac\azure/' failed with an unhandled exception:
    <ref *1> Error: invocation of azure-native:storage:listStorageAccountKeys returned an error: error reading from server: read tcp 127.0.0.1:57473->127.0.0.1:57471: wsarecv: An existing connection was forcibly closed by the remote host.
        at Object.callback (*\*\iac\node_modules\@pulumi\runtime\invoke.ts:261:37)
        at Object.onReceiveStatus (*\*\iac\node_modules\@grpc\grpc-js\src\client.ts:360:26)
        at Object.onReceiveStatus (*\*\iac\node_modules\@grpc\grpc-js\src\client-interceptors.ts:458:34)
        at Object.onReceiveStatus (*\*\iac\node_modules\@grpc\grpc-js\src\client-interceptors.ts:419:48)
        at *\*\iac\node_modules\@grpc\grpc-js\src\resolving-call.ts:163:24
        at processTicksAndRejections (node:internal/process/task_queues:77:11) {
      promise: Promise { <rejected> [Circular *1] }
    }

We upgraded these versions:

"@azure/arm-appcontainers": "^1.1.2", -> "^2.1.0"
"@azure/identity": "^3.3.0", -> "^4.4.1"
"@pulumi/azure": "^5.49.0", -> "^6.0.0"
"@pulumi/azure-native": "^1.104.0", -> "^2.63.0"
"@pulumi/azuread": "^5.40.0", -> "^5.53.4"
"@pulumi/pulumi": "^3.115.2", -> "^3.134.0"

Example

The source of the error seems to come from these lines:

const keys = await azure_native.storage.listStorageAccountKeys({
        accountName: storage_account_name,
        resourceGroupName: RESOURCE_GROUP_NAME
    });
    const firstKeyIndex = 0;
    const storage_account_access_key = keys.keys[firstKeyIndex].value;`

Output of `pulumi about

CLI
Version      3.115.0
Go Version   go1.22.2
Go Compiler  gc

Plugins
KIND      NAME          VERSION
resource  azure         6.0.0
resource  azure-native  2.63.0
resource  azuread       5.53.4
language  nodejs        unknown
resource  random        4.13.2

Host
OS       Microsoft Windows 11 Pro
Version  10.0.22631 Build 22631
Arch     x86_64

This project is written in nodejs: executable='C:\Program Files\nodejs\node.exe' version='v18.10.0'

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[thomas11]

Hi @TomMakes, sorry you're running into trouble here. Do I understand correctly that the same program worked before the dependency upgrades? Anything else you've updated?

Also, how is the azure native provider authenticating with Azure (Service Principal, MSI, OIDC, ...)?

Can you reproduce this problem in a minimal program that only uses Azure Native to create a storage account and list the keys?

[TomMakes]

Hi @TomMakes, sorry you're running into trouble here. Do I understand correctly that the same program worked before the dependency upgrades? Anything else you've updated?

Also, how is the azure native provider authenticating with Azure (Service Principal, MSI, OIDC, ...)?

Can you reproduce this problem in a minimal program that only uses Azure Native to create a storage account and list the keys?

Thanks for your reply. The same program worked before the upgrades. I don't believe I made other updates but it has been a month since last 'pulumi up' execution.

The azure native provider authenticates through az login in the CLI.

The past few days I've been building up a stack that just creates a storage account then calls listStorageAccountKeys. I was having errors even then, but right now I do have it working for me after I used pulumi.apply on the outputs to ensure they are given values before being used.

Here is my current code.

// Create an Azure Resource Group
const resourceGroup = { name: "test-rg" };

// Create an Azure resource (Storage Account)
const storageAccName = "storetest2";
const storageAccount = new storage.StorageAccount(storageAccName, {
    resourceGroupName: resourceGroup.name,
    accountName: storageAccName,
    sku: {
        name: storage.SkuName.Standard_LRS,
    },
    kind: storage.Kind.StorageV2,
});

const storageAccountKeys = pulumi.all(
  [resourceGroup.name, storageAccount.name]
  ).apply(([rgName, saName]) => {
    return storage.listStorageAccountKeys({
          resourceGroupName: rgName,
          accountName: saName
      });
  });

const firstKeyIndex = 0;
primaryStorageKey = pulumi.all([storageAccountKeys]).apply(([saK])=>{
  return `StringConcat with key: ${saK.keys[firstKeyIndex].value}`;
});

Now in my primary stack I'm getting a similar issue with a servicebus, so I'll be testing the creation of a service bus in my minimal stack with the storage account. Error I'm seeing for service bus:

Diagnostics:
  pulumi:pulumi:Stack (iac-dev):
    INFO: environment variables were configured using the local .env file

    panic: runtime error: index out of range [0] with length 0
    goroutine 26 [running]:
    github.com/pulumi/pulumi-azure-native/v2/provider/pkg/provider.(*azureNativeProvider).Diff(0xc0005d5888, {0x1d13a80?, 0x40?}, 0xc0007fe000)
        /home/runner/work/pulumi-azure-native/pulumi-azure-native/provider/pkg/provider/provider.go:809 +0x87b
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler.func1({0xdc76880?, 0xc00080bb30?}, {0x1ca1260?, 0xc0007fe000?})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.133.0/proto/go/provider_grpc.pb.go:633 +0xcb
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0xdc76880, 0xc00080b8c0}, {0x1ca1260, 0xc0007fe000}, 0xc000338880, 0xc00047a528)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3db
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler({0x1d13a80, 0xc0005d5888}, {0xdc76880, 0xc00080b8c0}, 0xc0002d4c80, 0xc0005a0d60)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.133.0/proto/go/provider_grpc.pb.go:635 +0x143
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc000816600, {0xdc76880, 0xc00080b830}, {0xdc800e0, 0xc00039a180}, 0xc0001b4900, 0xc000704450, 0xe592b40, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1369 +0xdf8
    google.golang.org/grpc.(*Server).handleStream(0xc000816600, {0xdc800e0, 0xc00039a180}, 0xc0001b4900)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1780 +0xe8b
    google.golang.org/grpc.(*Server).serveStreams.func2.1()
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1019 +0x8b
    created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 102
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1030 +0x125

  azure-native:servicebus:Namespace (devservicebus):
    error: error reading from server: read tcp 127.0.0.1:53849->127.0.0.1:53846: wsarecv: An existing connection was forcibly closed by the remote host.

[baoduy]

I got exactly the same issue. It very hard to fix as the error message very strange.

[thomas11]

Hi @TomMakes and @baoduy, it looks like this problem might be related to issues in the underlying pulumi/pulumi engine that have since been fixed. Could you try to upgrade the Azure Native provider to the latest 2.64.3 and try again?

[TomMakes]

I'm still getting the issue in my primary stack, however I'm not able to recreate it yet on my test stack.

[thomas11]

@TomMakes, can you double-check and report the versions of azure-native in both stacks?

[TomMakes]

Both versions are the same, azure-native 2.64.3

[thomas11]

We've now identified the root cause and it's a bug in the provider. #3633 fixed it and will be released shortly. Apologies for the inconvenience!

[thomas11]

Release v2.65.1 that just went out will fix at least the "index out of range" error you encountered in your attempt to reproduce the original issue.

[TomMakes]

That fixed the index out of range error!

[thomas11]

Great! Are you able to run your repro now? Does storage.listStorageAccountKeys fail?

[TomMakes]

"Great! Are you able to run your https://github.com/pulumi/pulumi-azure-native/issues/3597#issuecomment-2393909133 now? Does storage.listStorageAccountKeys fail?"

listStorageAccountKeys still looks like it needs pulumi.all/apply to give a valid output.

Below is how I currently am able to get primaryStorageKey returned as a defined value.

let primaryStorageKey;
const buildPulumiInfrastructure = async () => {
  // Create an Azure Resource Group
  const resourceGroup = { name: "pulumi-test-rg" };
  const RESOURCE_GROUP_NAME = resourceGroup.name;

  // Create an Azure resource (Storage Account)
  const storageAccName = "storetest2";
  const storageAccount = new azure_native.storage.StorageAccount(storageAccName, {
      resourceGroupName: resourceGroup.name,
      accountName: storageAccName,
      sku: {
          name: azure_native.storage.SkuName.Standard_LRS,
      },
      kind: azure_native.storage.Kind.StorageV2,
  });

  const storageAccountKeys = pulumi.all(
    [resourceGroup.name, storageAccount.name]
    ).apply(([rgName, saName]) => {
      return azure_native.storage.listStorageAccountKeys({
            resourceGroupName: rgName,
            accountName: saName
        });
    });

  const firstKeyIndex = 0;
  primaryStorageKey = pulumi.all([storageAccountKeys]).apply(([saK])=>{
    return `StringConcat with key: ${saK.keys[firstKeyIndex].value}`;
  });
}

buildPulumiInfrastructure().then(async () => {
    // console.log('done');
});

export const primaryStorageKeyConst = primaryStorageKey ?? 'Not defined';

[thomas11]

This is due to Pulumi's handling of outputs. To make it easier, however, there's also the output version of the function, listStorageAccountKeysOutput. It takes output arguments, so you can pass resourceGroup.name, storageAccount.name directly.

[thomas11]

@TomMakes, would you consider this issue resolved for you, or is there anything else we can do?

[TomMakes]

Yes I would consider this issue resolved, I didn't realize I needed to mark it. Thanks for the help @thomas11 !