Panic on successive updates of gcp.cloudrun.Service

cnunciato commented 2 weeks ago

Noticed this with a failing pulumi/examples test (apparently failing for a while, but still repros):

Given the following program:

import * as gcp from "@pulumi/gcp";

const _default = new gcp.cloudrun.Service("my-service", {
    project: "pulumi-development",
    location: "us-central1",
    template: {
        spec: {
            containers: [{
                image: "us-docker.pkg.dev/cloudrun/container/hello",
            }],
        },
    },
});

The first update succeeds, but successive updates fail consistently with the following output:

Updating (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/christian-pulumi-corp/gcp-typescript-1ce9a5c/dev/updates/6

     Type                 Name                        Status      Info
     Type                     Name                        Status         Info
     pulumi:pulumi:Stack      gcp-typescript-1ce9a5c-dev                 53 messages
     └─ gcp:cloudrun:Service  my-service                  **failed**     1 error

Diagnostics:
  gcp:cloudrun:Service (my-service):
    error: error reading from server: EOF

  pulumi:pulumi:Stack (gcp-typescript-1ce9a5c-dev):
    panic: value is null [recovered]
        panic: value is null
    goroutine 90 [running]:
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).recoverOnTypeError(0x14005bfc3c8?, {0x14004edaa50?, 0x104792ce0?}, {0x1045032e0?, 0x104f07540})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.85.0/pkg/tfbridge/provider.go:218 +0xf0
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.callWithRecover[...].func1()
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.85.0/pkg/tfbridge/provider.go:201 +0x64
    panic({0x1045032e0?, 0x104f07540?})
        /opt/hostedtoolcache/go/1.21.11/x64/src/runtime/panic.go:914 +0x218
    github.com/hashicorp/go-cty/cty.Value.LengthInt({{{0x104f37178?, 0x14005d481b0?}}, {0x0?, 0x0?}})
        /home/runner/go/pkg/mod/github.com/hashicorp/go-cty@v1.4.1-0.20200414143053-d3edf31b6320/cty/value_ops.go:989 +0x254
    github.com/hashicorp/go-cty/cty.Value.AsValueSlice({{{0x104f37178?, 0x14005d481b0?}}, {0x0?, 0x0?}})
        /home/runner/go/pkg/mod/github.com/hashicorp/go-cty@v1.4.1-0.20200414143053-d3edf31b6320/cty/value_ops.go:1216 +0x6c
    github.com/hashicorp/terraform-provider-google-beta/google-beta/tpgresource.SetMetadataLabelsDiff({0x0?, 0x0?}, 0x14005e16c80, {0x104eb8c00?, 0x14004ec2c00})
        /home/runner/work/pulumi-gcp/pulumi-gcp/upstream/google-beta/tpgresource/labels.go:172 +0x88
    github.com/hashicorp/terraform-provider-google-beta/google-beta/services/cloudrun.ResourceCloudRunService.All.func1({0x104f36aa8, 0x14005a77170}, 0x1039c2ccd?, {0x104eb8c00, 0x14004ec2c00})
        /home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/v2@v2.0.0-20240520223432-0c0bf0d65f10/helper/customdiff/compose.go:52 +0xa8
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.schemaMap.Diff(0x14000a22000, {0x104f36aa8, 0x14005a77170}, 0x14005d568f0, 0x14005d54500, 0x140007c81e0, {0x104eb8c00, 0x14004ec2c00}, 0x0)
        /home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/v2@v2.0.0-20240520223432-0c0bf0d65f10/helper/schema/schema.go:698 +0x3b8
    github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).SimpleDiff(0x0?, {0x104f36aa8?, 0x14005a77170?}, 0x14005d568f0, 0x14005cb0410?, {0x104eb8c00?, 0x14004ec2c00?})
        /home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/v2@v2.0.0-20240520223432-0c0bf0d65f10/helper/schema/resource.go:962 +0x9c
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.v2Provider.simpleDiff({0x104f36aa8?, {0x14005a77170?, 0x1039fd647?, 0x18?}}, {0x104f36aa8, 0x14005a77170}, 0x14005030b60?, 0x1006eece8?, 0x14005bfcf18?, {{{0x104f37258?, ...}}, ...}, ...)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.85.0/pkg/tfshim/sdk-v2/provider_diff.go:113 +0x298
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.v2Provider.Diff({0x140001d3020?, {0x140004bc560?, 0x10478d701?, 0x710001400506bea8?}}, {0x104f36aa8, 0x14005a77170}, {0x1039fd647, 0x18}, {0x104f37300?, 0x14005b03128}, ...)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.85.0/pkg/tfshim/sdk-v2/provider_diff.go:70 +0x27c
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.(*providerWithPlanResourceChangeDispatch).Diff(0x14000d08240, {0x104f36aa8, 0x14005a77170}, {0x1039fd647, 0x18}, {0x104f37300, 0x14005b03128}, {0x104f0fac0, 0x1400590d720}, {0x0?, ...})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.85.0/pkg/tfshim/sdk-v2/provider2.go:706 +0xfc
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Diff.func1()
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.85.0/pkg/tfbridge/provider.go:1064 +0x58
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.callWithRecover[...]({0x14004edaa50?, 0x14001460000?}, 0x100?, 0x104f3b960?)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.85.0/pkg/tfbridge/provider.go:206 +0x78
    github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Diff(0x14001460000, {0x104f36aa8?, 0x14005a76d50?}, 0x14005bc0680)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.85.0/pkg/tfbridge/provider.go:1063 +0x7d4
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.(*muxer).Diff.func1({0x104f4f8a8?, 0x14001460000?})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/muxer@v0.0.9-0.20240227144008-2da15b3d6f6e/muxer.go:369 +0x3c
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.resourceMethod[...](0x14004ceaf60?, 0x1400589d6c8, 0x1400589d6a8?)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/muxer@v0.0.9-0.20240227144008-2da15b3d6f6e/muxer.go:334 +0xc4
    github.com/pulumi/pulumi-terraform-bridge/x/muxer.(*muxer).Diff(0x1400589d708?, {0x104f36aa8?, 0x14005a76d50?}, 0x104575c00?)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/x/muxer@v0.0.9-0.20240227144008-2da15b3d6f6e/muxer.go:368 +0x5c
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler.func1({0x104f36aa8, 0x14005a76d50}, {0x104cca1e0?, 0x14005bc0680})
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.121.0/proto/go/provider_grpc.pb.go:633 +0x74
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x104f36aa8, 0x14005909da0}, {0x104cca1e0, 0x14005bc0680}, 0x14005bb7b40, 0x14005905068)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x2e8
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler({0x104e79f60?, 0x14004ceaf60}, {0x104f36aa8, 0x14005909da0}, 0x14005bc0600, 0x1400019cf40)
        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.121.0/proto/go/provider_grpc.pb.go:635 +0x12c
    google.golang.org/grpc.(*Server).processUnaryRPC(0x1400041d400, {0x104f36aa8, 0x14005909d10}, {0x104f47120, 0x1400039d500}, 0x140059157a0, 0x14004d56630, 0x109208000, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1369 +0xba0
    google.golang.org/grpc.(*Server).handleStream(0x1400041d400, {0x104f47120, 0x1400039d500}, 0x140059157a0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1780 +0xc80
    google.golang.org/grpc.(*Server).serveStreams.func2.1()
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1019 +0x8c
    created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 60
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/server.go:1030 +0x150

Resources:
    1 unchanged

Duration: 3s

Tested with TypeScript and Python.

tommyhe6 commented 1 week ago

Can confirm I'm getting this same error repeatedly with the TypeScript library for deploying Cloud Run. I'm new to Pulumi, but it seems to only occur in Github Actions for me and not when I run locally.

tommyhe6 commented 1 week ago

After some more testing, I've concluded:

If the Cloud Run service existed before, it fails with the above error. This happens with ubuntu-22.04, ubuntu-20.04, and macos-13. If the Cloud Run service did not exist before, it works as expected.
Locally, I have never gotten this issue and I am on macOS 13.5.2 22G91 arm64.

Would appreciate it if someone with more experience with Pulumi in Github Actions could take a look and see why it's happening!

VenelinMartinov commented 1 week ago

@cnunciato thanks for reporting and we'll take a look as soon as possible!

I wonder why the issue did not get a "needs-triage" label? Did you use one of the issue templates? I saw this by chance - without the label we might easily miss the issue.

I've confirmed this reproes in Python. Interestingly, I failed to repro in YAML.

mroloux commented 1 week ago

FYI: we're having the same issue. It's reproducible locally.

VenelinMartinov commented 1 week ago

Thanks for reporting and sorry you've hit this too.

I've reproduced this and am investigating the root-cause.

VenelinMartinov commented 1 week ago

OK, identified the issue and a fix is coming in the next release after https://github.com/pulumi/pulumi-gcp/pull/2165

As a workaround, anyone affected by this can specify a label for the metadata property like so:

"""A Google Cloud Python Pulumi program"""

import pulumi
import pulumi_gcp as gcp

gcp.cloudrun.Service(
    "my-service",
    location="us-central1",
    template=gcp.cloudrun.ServiceTemplateArgs(
        spec=gcp.cloudrun.ServiceTemplateSpecArgs(
            containers=[
                gcp.cloudrun.ServiceTemplateSpecContainerArgs(
                    image="gcr.io/cloudrun/hello:latest"
                )
            ]
        )
    ),
    metadata=gcp.cloudrun.ServiceMetadataArgs(
        labels={
            "mylabel": "myvalue"
        }
    )
)

This works around the issue as it is triggered by a null labels property inside the metadata.

pulumi-bot commented 1 week ago

This issue has been addressed in PR #2165 and shipped in release v7.30.1.

pulumi / pulumi-gcp

Panic on successive updates of gcp.cloudrun.Service #2155