search

pulumi / pulumi-aws

An Amazon Web Services (AWS) Pulumi resource package, providing multi-language access to AWS
Apache License 2.0
459 stars 156 forks source link

Panic from provider during an update #411

Closed ellismg closed 5 years ago

ellismg commented 5 years ago

We saw a panic during an update in this build: https://travis-ci.com/pulumi/examples/jobs/166367054

The output from the testing harness for the test was:

[ i/examples/cloud-js-containers ] sample: /home/travis/gopath/src/github.com/pulumi/examples/cloud-js-containers
[ i/examples/cloud-js-containers ] pulumi: /home/travis/.pulumi/bin/pulumi
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/aws-serverless to version 0.15.2-dev-1538607922-g002f527
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/cloud to version 0.16.3-dev.1545160257
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/kubernetes to version 0.18.1-dev.1545266618
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/openstack to version 0.16.4-dev.1545094127
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/terraform-template to version 0.15.0-dev-1534442432-gadc4bf5
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/azure to version 0.16.6-dev.1545108400
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/azure-serverless to version 0.15.4-dev.1545160136
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/cloud-azure to version 0.16.3-dev.1545160257
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/gcp to version 0.16.3-dev.1545094101
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/aws to version 0.16.5-dev.1545183793
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/random to version 0.2.1-dev.1545094178
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/vsphere to version 0.16.3-dev.1545094147
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/pulumi to version 0.16.9-dev.1545254396
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/aws-infra to version 0.16.3-dev.1545258948
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/cloud-aws to version 0.16.3-dev.1545160257
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/docker to version 0.16.3-dev.1544124266
[ i/examples/cloud-js-containers ] adding resolution for @pulumi/eks to version 0.16.5-dev.1544740879
[ i/examples/cloud-js-containers ] **** Invoke '/usr/local/bin/yarn install' in '/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765'
[ i/examples/cloud-js-containers ] Wrote output to /tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765/command-output/yarn-install.20181220-075518.bf023.log
[ i/examples/cloud-js-containers ] projdir: /tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765
[ i/examples/cloud-js-containers ] Initializing project (dir /tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765; stack p-it-travis-job-cloud-js-c-9fadf5a1)
[ i/examples/cloud-js-containers ] **** Invoke '/home/travis/.pulumi/bin/pulumi login --tracing https://tracing.pulumi-engineering.com/collector/api/v1/spans' in '/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765'
[ i/examples/cloud-js-containers ] Wrote output to /tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765/command-output/pulumi-login.20181220-075518.c4a4b.log
[ i/examples/cloud-js-containers ] **** Invoke '/home/travis/.pulumi/bin/pulumi stack init [secure]/p-it-travis-job-cloud-js-c-9fadf5a1 --tracing https://tracing.pulumi-engineering.com/collector/api/v1/spans' in '/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765'
[ i/examples/cloud-js-containers ] Wrote output to /tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765/command-output/pulumi-stack-init.20181220-075519.43d0b.log
[ i/examples/cloud-js-containers ] **** Invoke '/home/travis/.pulumi/bin/pulumi config set aws:region us-west-2 --tracing https://tracing.pulumi-engineering.com/collector/api/v1/spans' in '/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765'
[ i/examples/cloud-js-containers ] **** Invoke '/home/travis/.pulumi/bin/pulumi config set cloud-aws:useFargate true --tracing https://tracing.pulumi-engineering.com/collector/api/v1/spans' in '/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765'
[ i/examples/cloud-js-containers ] Performing primary preview and update
[ i/examples/cloud-js-containers ] **** Invoke '/home/travis/.pulumi/bin/pulumi preview --non-interactive --tracing https://tracing.pulumi-engineering.com/collector/api/v1/spans' in '/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765'
[ i/examples/cloud-js-containers ] Wrote output to /tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765/command-output/pulumi-preview-initial.20181220-075533.93abe.log
[ i/examples/cloud-js-containers ] **** Invoke '/home/travis/.pulumi/bin/pulumi up --non-interactive --skip-preview --tracing https://tracing.pulumi-engineering.com/collector/api/v1/spans' in '/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765'
[ i/examples/cloud-js-containers ] Invoke '/home/travis/.pulumi/bin/pulumi up --non-interactive --skip-preview --tracing https://tracing.pulumi-engineering.com/collector/api/v1/spans' failed: exit status 255
[ i/examples/cloud-js-containers ] Updating ([secure]/p-it-travis-job-cloud-js-c-9fadf5a1):
[ i/examples/cloud-js-containers ] 
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating 
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating flag provided but not defined: -tracing
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating Usage of tf-provider-flags:
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating   -get-provider-info
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating        dump provider info as JSON to stdout
[ i/examples/cloud-js-containers ]  +  aws-infra:network:Network default-vpc creating 
[ i/examples/cloud-js-containers ]  +  aws-infra:cluster:Cluster pulumi-p-it-travi-global creating 
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating 
[ i/examples/cloud-js-containers ]  +  cloud:global:infrastructure global-infrastructure creating 
[ i/examples/cloud-js-containers ]  +  aws:ecr:Repository pulum-164fa748-container creating 
[ i/examples/cloud-js-containers ]  +  aws:ecs:Cluster pulumi-p-it-travi-global creating 
[ i/examples/cloud-js-containers ]  +  aws:cloudwatch:LogGroup pulumi-nginx creating 
[ i/examples/cloud-js-containers ]  +  aws:iam:Role pulumi-p-it-tr-execution creating 
[ i/examples/cloud-js-containers ]  +  aws:iam:Role pulumi-p-it-travis--task creating 
[ i/examples/cloud-js-containers ]  +  aws:iam:Role pulumi-p-it-tr-execution created 
[ i/examples/cloud-js-containers ]  +  aws:ecr:Repository pulum-164fa748-container created 
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating Starting docker build and push...
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating Building container image 'pulum-164fa748-container': context=./app
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating Executing 'docker build ./app -t pulum-164fa748-container'
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating Sending build context to Docker daemon  17.92kB
[ i/examples/cloud-js-containers ]  +  aws:ec2:SecurityGroup pulumi-p-it-travi-global creating 
[ i/examples/cloud-js-containers ]  +  aws:elasticloadbalancingv2:TargetGroup 42dc3ff4 creating 
[ i/examples/cloud-js-containers ]  +  aws:iam:RolePolicyAttachment pulumi-p-it-tr-execution creating 
[ i/examples/cloud-js-containers ]  +  aws:ecr:LifecyclePolicy pulum-164fa748-container creating 
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating 
[ i/examples/cloud-js-containers ]  +  aws:ecs:Cluster pulumi-p-it-travi-global created 
[ i/examples/cloud-js-containers ]  +  aws:elasticloadbalancingv2:LoadBalancer 42dc3ff4 creating 
[ i/examples/cloud-js-containers ]  +  aws:iam:Role pulumi-p-it-travis--task created 
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating Step 1/2 : FROM nginx
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating latest: Pulling from library/nginx
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating a5a6f2f73cd8: Pulling fs layer
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating 33b176c904de: Download complete
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating 1ba02017c4b2: Verifying Checksum
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating 1ba02017c4b2: Download complete
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating a5a6f2f73cd8: Verifying Checksum
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating a5a6f2f73cd8: Pull complete
[ i/examples/cloud-js-containers ]  +  aws:iam:RolePolicyAttachment pulumi-p-i-task-fd1a00e5 creating 
[ i/examples/cloud-js-containers ]  +  aws:iam:RolePolicyAttachment pulumi-p-i-task-32be53a2 creating 
[ i/examples/cloud-js-containers ]  +  aws:cloudwatch:LogGroup pulumi-nginx created 
[ i/examples/cloud-js-containers ]  +  aws:iam:RolePolicyAttachment pulumi-p-it-tr-execution created 
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating panic: fatal: An assertion has failed: expected non-nil error with nil state during Create
[ i/examples/cloud-js-containers ]  +  aws:ecr:LifecyclePolicy pulum-164fa748-container creating error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  +  aws:iam:RolePolicyAttachment pulumi-p-i-task-32be53a2 creating error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  +  aws:elasticloadbalancingv2:LoadBalancer 42dc3ff4 creating error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  +  aws:iam:RolePolicyAttachment pulumi-p-i-task-fd1a00e5 creating error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  +  aws:ec2:SecurityGroup pulumi-p-it-travi-global creating error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  +  aws:elasticloadbalancingv2:TargetGroup 42dc3ff4 creating error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating goroutine 381 [running]:
[ i/examples/cloud-js-containers ]  +  aws:ecr:LifecyclePolicy pulum-164fa748-container **creating failed** error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  +  aws:iam:RolePolicyAttachment pulumi-p-i-task-32be53a2 **creating failed** error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  +  aws:elasticloadbalancingv2:LoadBalancer 42dc3ff4 **creating failed** error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  +  aws:iam:RolePolicyAttachment pulumi-p-i-task-fd1a00e5 **creating failed** error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  +  aws:ec2:SecurityGroup pulumi-p-it-travi-global **creating failed** error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  +  aws:elasticloadbalancingv2:TargetGroup 42dc3ff4 **creating failed** error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/pkg/util/contract.failfast(0xc000b18a00, 0x4c)
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating 1ba02017c4b2: Pull complete
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating    /home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/pkg/util/contract/failfast.go:23 +0xe9
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating 33b176c904de: Pull complete
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/pkg/util/contract.Assertf(0xc00011c000, 0x37bfcfe, 0x33, 0x0, 0x0, 0x0)
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating Digest: sha256:5d32f60db294b5deb55d078cd4feb410ad88e6fe77500c87d3970eca97f54dba
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating    /home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/pkg/util/contract/assert.go:33 +0x126
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating Status: Downloaded newer image for nginx:latest
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi-terraform/pkg/tfbridge.(*Provider).Create(0xc000012b40, 0x3af3780, 0xc000dbb740, 0xc000deda40, 0xc000012b40, 0x0, 0x0)
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating  ---> 568c4670fa80
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating    /home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi-terraform/pkg/tfbridge/provider.go:572 +0xbda
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating Step 2/2 : COPY content /usr/share/nginx/html
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Create_Handler.func1(0x3af3780, 0xc000dbb740, 0x356e020, 0xc000deda40, 0x3af3780, 0xc000dbb740, 0x3b06100, 0xc000ed40f0)
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating  ---> 0357895e970f
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating    /home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/sdk/proto/go/provider.pb.go:1229 +0x86
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating Successfully built 0357895e970f
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating github.com/pulumi/pulumi-aws/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x3af3780, 0xc000dbb740, 0x356e020, 0xc000deda40, 0xc00075a740, 0xc00075a760, 0x0, 0x0, 0x0, 0x0)
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating Successfully tagged pulum-164fa748-container:latest
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating    /home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57 +0x2b0
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating Executing 'docker image inspect -f {{.Id}} pulum-164fa748-container'
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Create_Handler(0x36489e0, 0xc000012b40, 0x3af3780, 0xc000dbb380, 0xc000c960f0, 0xc000391180, 0x0, 0x0, 0x0, 0x0)
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating sha256:0357895e970f86f68cd6f3b81a00867fcfbd4d87978838a360e6bcfbe360066d
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating    /home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/sdk/proto/go/provider.pb.go:1231 +0x158
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx creating Logging in to registry...
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc000104000, 0x3afb000, 0xc000104160, 0xc000eeb000, 0xc000556990, 0x666c3c0, 0x0, 0x0, 0x0)
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating error: Running program '/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765' failed with an unhandled exception:
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating error: Running program '/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765' failed with an unhandled exception:
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating error: Error: invocation of aws:ecr/getCredentials:getCredentials returned an error: all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:34803: connect: connection refused"
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating error: Error: invocation of aws:ecr/getCredentials:getCredentials returned an error: all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:34803: connect: connection refused"
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating error: Error: invocation of aws:ecr/getCredentials:getCredentials returned an error: all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:34803: connect: connection refused"
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating error: Error: invocation of aws:ecr/getCredentials:getCredentials returned an error: all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:34803: connect: connection refused"
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating error: Error: invocation of aws:ecr/getCredentials:getCredentials returned an error: all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:34803: connect: connection refused"
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating error: Error: invocation of aws:ecr/getCredentials:getCredentials returned an error: all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:34803: connect: connection refused"
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 creating error: Error: invocation of aws:ecr/getCredentials:getCredentials returned an error: all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:34803: connect: connection refused"
[ i/examples/cloud-js-containers ]  +  pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1 **creating failed** 2 errors; 26 messages
[ i/examples/cloud-js-containers ]  +  aws-infra:cluster:Cluster pulumi-p-it-travi-global created 
[ i/examples/cloud-js-containers ]  +  cloud:global:infrastructure global-infrastructure created 
[ i/examples/cloud-js-containers ]  +  aws-infra:network:Network default-vpc created 
[ i/examples/cloud-js-containers ]  +  cloud:service:Service pulumi-nginx created 
[ i/examples/cloud-js-containers ]  
[ i/examples/cloud-js-containers ] Diagnostics:
[ i/examples/cloud-js-containers ]   pulumi:pulumi:Stack (container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1):
[ i/examples/cloud-js-containers ]     flag provided but not defined: -tracing
[ i/examples/cloud-js-containers ]     Usage of tf-provider-flags:
[ i/examples/cloud-js-containers ]       -get-provider-info
[ i/examples/cloud-js-containers ]          dump provider info as JSON to stdout
[ i/examples/cloud-js-containers ]     panic: fatal: An assertion has failed: expected non-nil error with nil state during Create
[ i/examples/cloud-js-containers ]     goroutine 381 [running]:
[ i/examples/cloud-js-containers ]     github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/pkg/util/contract.failfast(0xc000b18a00, 0x4c)
[ i/examples/cloud-js-containers ]      /home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/pkg/util/contract/failfast.go:23 +0xe9
[ i/examples/cloud-js-containers ]     github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/pkg/util/contract.Assertf(0xc00011c000, 0x37bfcfe, 0x33, 0x0, 0x0, 0x0)
[ i/examples/cloud-js-containers ]      /home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/pkg/util/contract/assert.go:33 +0x126
[ i/examples/cloud-js-containers ]     github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi-terraform/pkg/tfbridge.(*Provider).Create(0xc000012b40, 0x3af3780, 0xc000dbb740, 0xc000deda40, 0xc000012b40, 0x0, 0x0)
[ i/examples/cloud-js-containers ]      /home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi-terraform/pkg/tfbridge/provider.go:572 +0xbda
[ i/examples/cloud-js-containers ]     github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Create_Handler.func1(0x3af3780, 0xc000dbb740, 0x356e020, 0xc000deda40, 0x3af3780, 0xc000dbb740, 0x3b06100, 0xc000ed40f0)
[ i/examples/cloud-js-containers ]      /home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/sdk/proto/go/provider.pb.go:1229 +0x86
[ i/examples/cloud-js-containers ]     github.com/pulumi/pulumi-aws/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x3af3780, 0xc000dbb740, 0x356e020, 0xc000deda40, 0xc00075a740, 0xc00075a760, 0x0, 0x0, 0x0, 0x0)
[ i/examples/cloud-js-containers ]      /home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57 +0x2b0
[ i/examples/cloud-js-containers ]     github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/sdk/proto/go._ResourceProvider_Create_Handler(0x36489e0, 0xc000012b40, 0x3af3780, 0xc000dbb380, 0xc000c960f0, 0xc000391180, 0x0, 0x0, 0x0, 0x0)
[ i/examples/cloud-js-containers ]      /home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/github.com/pulumi/pulumi/sdk/proto/go/provider.pb.go:1231 +0x158
[ i/examples/cloud-js-containers ]     github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc000104000, 0x3afb000, 0xc000104160, 0xc000eeb000, 0xc000556990, 0x666c3c0, 0x0, 0x0, 0x0)
[ i/examples/cloud-js-containers ]      /home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc/server.go:839 +0x8ec
[ i/examples/cloud-js-containers ]     github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000104000, 0x3afb000, 0xc000104160, 0xc000eeb000, 0x0)
[ i/examples/cloud-js-containers ]      /home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc/server.go:1036 +0x1308
[ i/examples/cloud-js-containers ]     github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00003c2b0, 0xc000104000, 0x3afb000, 0xc000104160, 0xc000eeb000)
[ i/examples/cloud-js-containers ]      /home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc/server.go:585 +0x9f
[ i/examples/cloud-js-containers ]     created by github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
[ i/examples/cloud-js-containers ]      /home/travis/gopath/src/github.com/pulumi/pulumi-aws/vendor/google.golang.org/grpc/server.go:583 +0xa1
[ i/examples/cloud-js-containers ]  
[ i/examples/cloud-js-containers ]     error: Running program '/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765' failed with an unhandled exception:
[ i/examples/cloud-js-containers ]     error: Error: invocation of aws:ecr/getCredentials:getCredentials returned an error: all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:34803: connect: connection refused"
[ i/examples/cloud-js-containers ]         at monitor.invoke (/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765/node_modules/@pulumi/cloud-aws/node_modules/@pulumi/aws/node_modules/@pulumi/pulumi/runtime/invoke.js:73:33)
[ i/examples/cloud-js-containers ]         at Object.onReceiveStatus (/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765/node_modules/grpc/src/client_interceptors.js:1205:9)
[ i/examples/cloud-js-containers ]         at InterceptingListener._callNext (/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765/node_modules/grpc/src/client_interceptors.js:568:42)
[ i/examples/cloud-js-containers ]         at InterceptingListener.onReceiveStatus (/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765/node_modules/grpc/src/client_interceptors.js:618:8)
[ i/examples/cloud-js-containers ]         at callback (/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765/node_modules/grpc/src/client_interceptors.js:845:24)
[ i/examples/cloud-js-containers ]  
[ i/examples/cloud-js-containers ]   aws:elasticloadbalancingv2:LoadBalancer (42dc3ff4):
[ i/examples/cloud-js-containers ]     error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  
[ i/examples/cloud-js-containers ]   aws:iam:RolePolicyAttachment (pulumi-p-i-task-fd1a00e5):
[ i/examples/cloud-js-containers ]     error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  
[ i/examples/cloud-js-containers ]   aws:elasticloadbalancingv2:TargetGroup (42dc3ff4):
[ i/examples/cloud-js-containers ]     error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  
[ i/examples/cloud-js-containers ]   aws:ec2:SecurityGroup (pulumi-p-it-travi-global):
[ i/examples/cloud-js-containers ]     error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  
[ i/examples/cloud-js-containers ]   aws:iam:RolePolicyAttachment (pulumi-p-i-task-32be53a2):
[ i/examples/cloud-js-containers ]     error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  
[ i/examples/cloud-js-containers ]   aws:ecr:LifecyclePolicy (pulum-164fa748-container):
[ i/examples/cloud-js-containers ]     error: Plan apply failed: transport is closing
[ i/examples/cloud-js-containers ]  
[ i/examples/cloud-js-containers ] Resources:
[ i/examples/cloud-js-containers ]     + 11 created
[ i/examples/cloud-js-containers ] 
[ i/examples/cloud-js-containers ] Duration: 13s
[ i/examples/cloud-js-containers ] 
[ i/examples/cloud-js-containers ] Permalink: https://app.pulumi.com/[secure]/p-it-travis-job-cloud-js-c-9fadf5a1/updates/1
[ i/examples/cloud-js-containers ] error: update failed
[ i/examples/cloud-js-containers ] 
[ i/examples/cloud-js-containers ] Wrote output to /tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765/command-output/pulumi-update-initial.20181220-075550.d620b.log
[ i/examples/cloud-js-containers ] Destroying stack
[ i/examples/cloud-js-containers ] **** Invoke '/home/travis/.pulumi/bin/pulumi destroy --non-interactive --skip-preview --tracing https://tracing.pulumi-engineering.com/collector/api/v1/spans' in '/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765'
[ i/examples/cloud-js-containers ] Still running command '/home/travis/.pulumi/bin/pulumi destroy --non-interactive --skip-preview --tracing https://tracing.pulumi-engineering.com/collector/api/v1/spans' (/tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765)...
[ i/examples/cloud-js-containers ] Invoke '/home/travis/.pulumi/bin/pulumi destroy --non-interactive --skip-preview --tracing https://tracing.pulumi-engineering.com/collector/api/v1/spans' failed: exit status 255
[ i/examples/cloud-js-containers ] Destroying ([secure]/p-it-travis-job-cloud-js-c-9fadf5a1):
[ i/examples/cloud-js-containers ]     pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1  flag provided but not defined: -tracing
[ i/examples/cloud-js-containers ]     pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1  Usage of tf-provider-flags:
[ i/examples/cloud-js-containers ]     pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1    -get-provider-info
[ i/examples/cloud-js-containers ]     pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1         dump provider info as JSON to stdout
[ i/examples/cloud-js-containers ] 
[ i/examples/cloud-js-containers ]  -  aws:iam:RolePolicyAttachment pulumi-p-it-tr-execution deleting 
[ i/examples/cloud-js-containers ]  -  aws:iam:RolePolicyAttachment pulumi-p-it-tr-execution deleted 
[ i/examples/cloud-js-containers ]  -  aws:cloudwatch:LogGroup pulumi-nginx deleting 
[ i/examples/cloud-js-containers ]  -  aws:ecs:Cluster pulumi-p-it-travi-global deleting 
[ i/examples/cloud-js-containers ]  -  aws:iam:Role pulumi-p-it-tr-execution deleting 
[ i/examples/cloud-js-containers ]  -  aws:iam:Role pulumi-p-it-travis--task deleting 
[ i/examples/cloud-js-containers ]  -  aws:cloudwatch:LogGroup pulumi-nginx deleted 
[ i/examples/cloud-js-containers ]  -  aws:iam:Role pulumi-p-it-tr-execution deleted 
[ i/examples/cloud-js-containers ]  -  aws:ecs:Cluster pulumi-p-it-travi-global deleted 
[ i/examples/cloud-js-containers ] @ Destroying....
[ i/examples/cloud-js-containers ]  -  aws:iam:Role pulumi-p-it-travis--task deleting error: Plan apply failed: deleting urn:pulumi:p-it-travis-job-cloud-js-c-9fadf5a1::container-quickstart::cloud:global:infrastructure$aws:iam/role:Role::pulumi-p-it-travis--task: DeleteConflict: Cannot delete entity, must detach all policies first.
[ i/examples/cloud-js-containers ]  -  aws:iam:Role pulumi-p-it-travis--task **deleting failed** error: Plan apply failed: deleting urn:pulumi:p-it-travis-job-cloud-js-c-9fadf5a1::container-quickstart::cloud:global:infrastructure$aws:iam/role:Role::pulumi-p-it-travis--task: DeleteConflict: Cannot delete entity, must detach all policies first.
[ i/examples/cloud-js-containers ]     pulumi:pulumi:Stack container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1  4 messages
[ i/examples/cloud-js-containers ]  
[ i/examples/cloud-js-containers ] Diagnostics:
[ i/examples/cloud-js-containers ]   pulumi:pulumi:Stack (container-quickstart-p-it-travis-job-cloud-js-c-9fadf5a1):
[ i/examples/cloud-js-containers ]     flag provided but not defined: -tracing
[ i/examples/cloud-js-containers ]     Usage of tf-provider-flags:
[ i/examples/cloud-js-containers ]       -get-provider-info
[ i/examples/cloud-js-containers ]          dump provider info as JSON to stdout
[ i/examples/cloud-js-containers ]  
[ i/examples/cloud-js-containers ]   aws:iam:Role (pulumi-p-it-travis--task):
[ i/examples/cloud-js-containers ]     error: Plan apply failed: deleting urn:pulumi:p-it-travis-job-cloud-js-c-9fadf5a1::container-quickstart::cloud:global:infrastructure$aws:iam/role:Role::pulumi-p-it-travis--task: DeleteConflict: Cannot delete entity, must detach all policies first.
[ i/examples/cloud-js-containers ]      status code: 409, request id: bd987cf8-042c-11e9-9a74-8d9c0ba14f29
[ i/examples/cloud-js-containers ]  
[ i/examples/cloud-js-containers ] Resources:
[ i/examples/cloud-js-containers ]     - 4 deleted
[ i/examples/cloud-js-containers ] 
[ i/examples/cloud-js-containers ] Duration: 33s
[ i/examples/cloud-js-containers ] 
[ i/examples/cloud-js-containers ] Permalink: https://app.pulumi.com/[secure]/p-it-travis-job-cloud-js-c-9fadf5a1/updates/2
[ i/examples/cloud-js-containers ] error: update failed
[ i/examples/cloud-js-containers ] 
[ i/examples/cloud-js-containers ] Wrote output to /tmp/p-it-travis-job-cloud-js-c-9fadf5a1-271188765/command-output/pulumi-destroy.20181220-075624.27fa2.log

I suspect the failure in delete was because we orphaned a resource while panicing during the update and did not know to delete it from the checkpoint, but I'm actually not sure if that's an issue as well (I thought we would have recorded that we were going to do the action before preforming it and the checkpoint would end up in an indeterminate state and the destroy operation would fail to start, but I am not sure).

lukehoban commented 5 years ago

Closing this out as we haven't seen it again - if we see it multiple times or hear a user report we can investigate further.