gcp.compute.ForwardingRule breaks on latest pulumi/gcp 7.38.0 & pulumi/gcp 8.3.1

[cl3nn]

Describe what happened

When we update the Pulumi GCP provider from version 6 to either version 7 (7.38.0) or version 8 (8.3.1), we encounter an unusual error with the ForwardingRule resource during the pulumi preview step:

    error:   sdk-v2/provider2.go:509: sdk.helper_schema: missing expected {: provider=google-beta@8.3.1
    error: diffing urn:pulumi:dev::web-3-loadbalancer::avrotros:InternalLoadBalancer$gcp:compute/forwardingRule:ForwardingRule::internal-lb-fw-rule: 1 error occurred:
        * [service_directory_registrations[0]] missing expected {

Everything works correctly when using Pulumi GCP version 6.67.1."

Sample program

problematic resource inside a ComponentResource:

        this.forwardingRule = new gcp.compute.ForwardingRule(
            `${name}-fw-rule`,
            {
                target: this.regionHttpProxy.id,
                portRange: "80",
                region: args.region,
                loadBalancingScheme: "INTERNAL_MANAGED",
                network: args.networkId,
                subnetwork: this.subNetwork.id,
                allowGlobalAccess: true,
            },
            { parent: this },
        );

Log output

when using pulumi/gcp 8.3.1

    error:   sdk-v2/provider2.go:509: sdk.helper_schema: missing expected {: provider=google-beta@8.3.1
    error: diffing urn:pulumi:dev::web-3-loadbalancer::avrotros:InternalLoadBalancer$gcp:compute/forwardingRule:ForwardingRule::internal-lb-fw-rule: 1 error occurred:
        * [service_directory_registrations[0]] missing expected {

when using pulumi/gcp 7.38.0

    error:   sdk-v2/provider2.go:385: sdk.helper_schema: missing expected {: provider=google-beta@7.38.0
    error: diffing urn:pulumi:dev::web-3-loadbalancer::avrotros:InternalLoadBalancer$gcp:compute/forwardingRule:ForwardingRule::internal-lb-fw-rule: 1 error occurred:
        * [service_directory_registrations[0]] missing expected {

Affected Resource(s)

No response

Output of pulumi about

when using pulumi/gcp 8.3.1

Version      3.134.1
Go Version   go1.23.1
Go Compiler  gc

Plugins
KIND      NAME    VERSION
language  nodejs  unknown

Host     
OS       ubuntu
Version  22.04
Arch     x86_64

This project is written in nodejs: executable='/home/clenn/.nvm/versions/node/v20.11.1/bin/node' version='v20.11.1'

when using pulumi/gcp 7.38.0

Version      3.134.1
Go Version   go1.23.1
Go Compiler  gc

Plugins
KIND      NAME    VERSION
resource  gcp     7.38.0
language  nodejs  unknown

Host     
OS       ubuntu
Version  22.04
Arch     x86_64

This project is written in nodejs: executable='/home/clenn/.nvm/versions/node/v20.11.1/bin/node' version='v20.11.1'`

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[guineveresaenger]

Hi @cl3nn, thank you for reporting this. We're sorry you're having trouble.

The likely cause for what you're seeing is that between these versions, the schema changed behavior slightly. Usually we would see this right away in the code itself and you'd be able to change it there; however it looks like this is affecting a provider default setting.

Could you please provide us with the following:

• a gist of the pulumi stack state before the upgrade? (pulumi stack import --file stack.json)
• a minimal runnable program that reproduces this behavior, to assist us debugging on our end?

Thank you so much!

[cl3nn]

Hi @guineveresaenger i cannot give you the complete stack state due to security reasons, but i've a redacted state of the working resource before the upgrade here: https://gist.github.com/cl3nn/7d05759a43250567beb4a0f2c5515641

A minimal resource was given, based on this example: https://www.pulumi.com/registry/packages/gcp/api-docs/compute/forwardingrule/#forwarding-rule-http-lb

[guineveresaenger]

hey @cl3nn - okay I think I have a workaround for you. 🤞

I managed to reproduce it on a smallerloadbalancer example; might be handy for testing.

Basically, what's happening is that v6 --> v7, the serviceDirectoryRegistrations type changed from a list with a single item to an object. For me, upgrading from v6.67.1 (newest v6) --> v7.38.0 (the version you tried to upgrade to), I could reproduce the error. When you recreate the stack from scratch using v7, the stack file reads "serviceDirectoryRegistrations": null, not "serviceDirectoryRegistrations": [],, like it does for v6.

This is all especially frustrating of course for a user who isn't even using this field!

We do have means to prevent this sort of thing. It appears as though at least at some point we (or probably the upstream provider) had a working state migration from v6 --> v7, which was lost by the time v7.38.0 was published. I discovered this because I initially attempted to upgrade v6.67.1 --> v7.0.0, which succeeds. The state for the offending field is updated to be the expected null object.

In summary:

You should be able to upgrade today, without any changes, if you do it in two steps, avoiding v7.38.0:

1. upgrade v6.x --> v7.0.0
2. then upgrade --> v8.x

We will also very likely fix this universally to make sure we catch all state upgrades for everyone.