search

pulumi / pulumi-aws

An Amazon Web Services (AWS) Pulumi resource package, providing multi-language access to AWS
Apache License 2.0
460 stars 155 forks source link

Panic when updating an aws.ssm.Parameter value #2017

Closed cnunciato closed 8 months ago

cnunciato commented 2 years ago

Given the following program:

import * as aws from "@pulumi/aws";

const someParam = new aws.ssm.Parameter("some-param", {
    name: "/cnunciato/shared-config-ts/dev/someParam",
    value: "some-value",
    type: "String",
});

... the initial update succeeds, but subsequent updates that change the parameter's value property fail:

$ pulumi up
...

Updating (dev)

View Live: https://app.pulumi.com/cnunciato/shared-config-ts/dev/updates/3

     Type                  Name                  Status                  Info
     pulumi:pulumi:Stack   shared-config-ts-dev  **failed**              1 error; 28 messages
 ~   └─ aws:ssm:Parameter  some-param            **updating failed**     [diff: ~value]; 1 error

Diagnostics:
  aws:ssm:Parameter (some-param):
    error: error reading from server: EOF

  pulumi:pulumi:Stack (shared-config-ts-dev):
    panic: interface conversion: interface {} is nil, not map[string]interface {}
    goroutine 64 [running]:
    github.com/hashicorp/go-cty/cty.Value.GetAttr({{{0xb8be390?, 0xc0005d0930?}}, {0x0?, 0x0?}}, {0xaab1393, 0x4})
        /home/runner/go/pkg/mod/github.com/hashicorp/go-cty@v1.4.1-0.20200414143053-d3edf31b6320/cty/value_ops.go:711 +0x2ed
    github.com/hashicorp/terraform-provider-aws/internal/service/ssm.resourceParameterUpdate(0xc000d3b500, {0x95882a0?, 0xc00068e000})
        /home/runner/go/pkg/mod/github.com/pulumi/terraform-provider-aws@v1.38.1-0.20220621164749-d0dee9d3594a/internal/service/ssm/parameter.go:282 +0x3e9
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0xb8bd5c8?, {0xb8bd5c8?, 0xc0000740b8?}, 0xd?, {0x95882a0?, 0xc00068e000?})
        /home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/v2@v2.0.0-20220505215311-795430389fa7/helper/schema/resource.go:719 +0x178
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc001673860, {0xb8bd5c8, 0xc0000740b8}, 0xc00172f5f0, 0xc000d3b080, {0x95882a0, 0xc00068e000})
        /home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/v2@v2.0.0-20220505215311-795430389fa7/helper/schema/resource.go:837 +0x605
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.v2Provider.Apply({0xc000d42000?}, {0xaafc3ae, 0x11}, {0xb8be438?, 0xc00188c920}, {0xb8c3ab0?, 0xc000d3b080})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.24.1/pkg/tfshim/sdk-v2/provider.go:112 +0x19b
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Update(0xc000246fc0, {0xb8bd638?, 0xc001e71e60?}, 0xc000d3a780)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.24.1/pkg/tfbridge/provider.go:1052 +0x86f
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Update_Handler.func1({0xb8bd638, 0xc001e71e60}, {0xa80a860?, 0xc000d3a780})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.33.1/proto/go/provider.pb.go:2650 +0x7b
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0xb8bd638, 0xc001e71a70}, {0xa80a860, 0xc000d3a780}, 0xc0019c4940, 0xc000d36450)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3f9
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Update_Handler({0xa945000?, 0xc000246fc0}, {0xb8bd638, 0xc001e71a70}, 0xc001e365a0, 0xc0018f0f00)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.33.1/proto/go/provider.pb.go:2652 +0x138
    google.golang.org/grpc.(*Server).processUnaryRPC(0xc0006fce00, {0xb8c3ce8, 0xc001357a00}, 0xc001e786c0, 0xc001a2ae70, 0x124640d0, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:1283 +0xcfd
    google.golang.org/grpc.(*Server).handleStream(0xc0006fce00, {0xb8c3ce8, 0xc001357a00}, 0xc001e786c0, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:1620 +0xa1b
    google.golang.org/grpc.(*Server).serveStreams.func1.2()
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:922 +0x98
    created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:920 +0x28a

    error: update failed

Resources:
    1 unchanged

Repros with other languages as well. (Initially noticed with YAML.) Does not repro in TF.

stack72 commented 2 years ago

Hi @cnunciato

We are working on https://github.com/pulumi/pulumi-terraform-bridge/issues/548 to fix this up!

This is also a case of needing work because in v4.20.0 of terraform-provider-aws, LaunchConfig changed to use the GetRawConfig which would have caused panics any time that resource was used

Paul

afa4 commented 2 years ago

Is there any workaround to bypass this problem, while fix is not deployed?

cnunciato commented 2 years ago

@afa4 Apologies for the delay here, the team's actively working to get this fixed. The current workaround is to downgrade to a previous version of the AWS provider. It looks like 5.8.0 introduced the issue, and I've confirmed that 5.7.2 works, so depending on which language you're using, you'll want to do something like this (e.g., for Node.js):

npm install @pulumi/aws@5.7.2
pulumi up

Example run with repro and correction for reference:

➜  pulumi up
Previewing update (dev)

     Type                  Name     Plan       Info
     pulumi:pulumi:Stack   ssm-dev             
 ~   └─ aws:ssm:Parameter  param    update     [diff: ~value]

Resources:
    ~ 1 to update
    1 unchanged

Do you want to perform this update? yes
Updating (dev)

     Type                  Name     Status                  Info
     pulumi:pulumi:Stack   ssm-dev  **failed**              1 error; 28 messages
 ~   └─ aws:ssm:Parameter  param    **updating failed**     [diff: ~value]; 1 error

Diagnostics:
  pulumi:pulumi:Stack (ssm-dev):
    panic: interface conversion: interface {} is nil, not map[string]interface {}
    ...
    error: update failed

  aws:ssm:Parameter (param):
    error: error reading from server: EOF

Resources:
    1 unchanged

Duration: 2s

➜  npm install @pulumi/aws@5.7.2

changed 1 package, and audited 117 packages in 2s

➜  pulumi up                    
Previewing update (dev)

     Type                  Name     Plan       Info
     pulumi:pulumi:Stack   ssm-dev             
 ~   └─ aws:ssm:Parameter  param    update     [diff: ~value]

Resources:
    ~ 1 to update
    1 unchanged

Do you want to perform this update? yes
Updating (dev)

     Type                  Name     Status      Info
     pulumi:pulumi:Stack   ssm-dev              
 ~   └─ aws:ssm:Parameter  param    updated     [diff: ~value]

Resources:
    ~ 1 updated
    1 unchanged

Duration: 4s
AleksandrAksenov commented 2 years ago

Hi

For python downgrade from 5.9.2 to 5.7.2 doesn't work properly.

error: Program failed with an unhandled exception:
error: Traceback (most recent call last):
  File "/opt/homebrew/bin/pulumi-language-python-exec", line 107, in <module>
    loop.run_until_complete(coro)
  File "/opt/homebrew/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "./venv/lib/python3.9/site-packages/pulumi/runtime/stack.py", line 126, in run_in_stack
    await run_pulumi_func(lambda: Stack(func))
  File "./venv/lib/python3.9/site-packages/pulumi/runtime/stack.py", line 49, in run_pulumi_func
    func()
  File "./venv/lib/python3.9/site-packages/pulumi/runtime/stack.py", line 126, in <lambda>
    await run_pulumi_func(lambda: Stack(func))
  File "./venv/lib/python3.9/site-packages/pulumi/runtime/stack.py", line 149, in __init__
    func()
  File "/opt/homebrew/bin/pulumi-language-python-exec", line 106, in <lambda>
    coro = pulumi.runtime.run_in_stack(lambda: runpy.run_path(args.PROGRAM, run_name='__main__'))
  File "/opt/homebrew/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 305, in run_path
    return _run_code(code, mod_globals, init_globals,
  File "/opt/homebrew/Cellar/python@3.9/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "./aws/app-api/./__main__.py", line 32, in <module>
    aws_caller_identity = aws.get_caller_identity()
  File "./venv/lib/python3.9/site-packages/pulumi_aws/get_caller_identity.py", line 103, in get_caller_identity
    __ret__ = pulumi.runtime.invoke('aws:index/getCallerIdentity:getCallerIdentity', __args__, opts=opts, typ=GetCallerIdentityResult).value
  File "./venv/lib/python3.9/site-packages/pulumi/runtime/invoke.py", line 166, in invoke
    raise invoke_error
Exception: invoke of aws:index/getCallerIdentity:getCallerIdentity failed: invocation of aws:index/getCallerIdentity:getCallerIdentity returned an error: unable to validate AWS AccessKeyID and/or SecretAccessKey - see https://pulumi.io/install/aws.html for details on configuration

error: an unhandled error occurred: Program exited with non-zero exit code: 1
AleksandrAksenov commented 2 years ago

For python version 5.10.0 works perfectly.

patwritescode commented 2 years ago

5.10.0 fixed this for me for node/typescript as well. much appreciated.

BernalCarlos commented 1 year ago

This is happening for me on version 5.35.0

josevromo commented 1 year ago

Happening with "@pulumi/aws": "^5.42.0"

epeters3 commented 9 months ago

Happening with @pulumi/aws@6.2.1.

epeters3 commented 9 months ago

I set overwrite: true [1] on the aws.ssm.Parameter definition and that solved the problem: I was able to update.

epeters3 commented 9 months ago

Scratch that. When I tried a second update with overwrite: true, I got the error again.

iwahbe commented 9 months ago

@epeters3 Can you post a program that replicates this panic on the most recent pulumi-aws version? I tried to repro, but I don't see a panic.

t0yv0 commented 8 months ago 

t0yv0@Antons-MacBook-Pro> pulumi up --yes                                                                                                                                                                                                                                                                                                                                                ~/tmp/2024/03/aws-2017
Previewing update (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/anton-pulumi-corp/aws-2017/dev/previews/316fd999-a2be-4c0c-b135-db2461291d9a

     Type                 Name          Plan     
     pulumi:pulumi:Stack  aws-2017-dev           

Resources:
    2 unchanged

Updating (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/anton-pulumi-corp/aws-2017/dev/updates/2

     Type                 Name          Status     
     pulumi:pulumi:Stack  aws-2017-dev             

Resources:
    2 unchanged

Duration: 4s

CLI          
Version      3.99.0
Go Version   go1.21.5
Go Compiler  gc

Plugins
NAME    VERSION
aws     6.24.2
awsx    2.5.0
docker  4.5.1
docker  3.6.1
nodejs  unknown

Host     
OS       darwin
Version  14.2.1
Arch     x86_64

This project is written in nodejs: executable='/Users/t0yv0/bin/node' version='v18.18.2'

Current Stack: anton-pulumi-corp/aws-2017/dev

TYPE                         URN
pulumi:pulumi:Stack          urn:pulumi:dev::aws-2017::pulumi:pulumi:Stack::aws-2017-dev
pulumi:providers:aws         urn:pulumi:dev::aws-2017::pulumi:providers:aws::default_6_24_2
aws:ssm/parameter:Parameter  urn:pulumi:dev::aws-2017::aws:ssm/parameter:Parameter::some-param

Found no pending operations associated with dev

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/anton-pulumi-corp
User           anton-pulumi-corp
Organizations  anton-pulumi-corp, moolumi, pulumi
Token type     personal

Dependencies:
NAME            VERSION
@pulumi/aws     6.24.2
@pulumi/awsx    2.5.0
@pulumi/pulumi  3.108.1
@types/node     18.19.21

Pulumi locates its logs in /var/folders/gk/cchgxh512m72f_dmkcc3d09h0000gp/T/com.apple.shortcuts.mac-helper// by default
t0yv0 commented 8 months ago

Folks I think we should close this one because it works as stated, but let's open new issues with repro programs if anything is still not working as expected. Thanks a lot!