GKE Autopilot is recreated without any changes to node_config

[ocervell]

Describe what happened

GKE Autopilot is recreated without any changes to node_config.

Running pulumi up twice shows on the second up that the cluster needs to be replaced because nodeConfig is different:

gcp:container:Cluster                    gke-cluster            replace     [diff: ~nodeConfig]

Sample program

import pulumi
import pulumi_gcp as gcp

config = pulumi.Config()
provider_cfg = pulumi.Config("gcp")
project = provider_cfg.require('project')
region = provider_cfg.get('region', 'europe-west1')
cluster = gcp.container.Cluster("gke-cluster",
    location=region,
    enable_autopilot=True,
    node_config={
        "oauth_scopes": ["https://www.googleapis.com/auth/cloud-platform"]
    }
)

Log output

No response

Affected Resource(s)

No response

Output of pulumi about

CLI          
Version      3.127.0
Go Version   go1.22.5
Go Compiler  gc

Plugins
KIND      NAME        VERSION
resource  gcp         7.32.0
resource  kubernetes  4.15.0
language  python      unknown

Host     
OS       debian
Version  12.4
Arch     x86_64

This project is written in python: executable='/home/osboxes/Workspace/pulumi-scripts/internal/venv/bin/python3' version='3.11.2'

Current Stack: ocervell/internal/freelabz-dev

TYPE                                     URN
pulumi:pulumi:Stack                      urn:pulumi:freelabz-dev::internal::pulumi:pulumi:Stack::internal-freelabz-dev
pulumi:providers:gcp                     urn:pulumi:freelabz-dev::internal::pulumi:providers:gcp::default_7_32_0
gcp:compute/globalAddress:GlobalAddress  urn:pulumi:freelabz-dev::internal::gcp:compute/globalAddress:GlobalAddress::app-freelabz
gcp:container/cluster:Cluster            urn:pulumi:freelabz-dev::internal::gcp:container/cluster:Cluster::gke-cluster
pulumi:providers:kubernetes              urn:pulumi:freelabz-dev::internal::pulumi:providers:kubernetes::gke_k8s

Found no pending operations associated with freelabz-dev

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/ocervell
User           ocervell
Organizations  ocervell
Token type     personal

Dependencies:
NAME                VERSION
cryptography        43.0.0
fastapi             0.111.1
httptools           0.6.1
pip                 24.1.2
pulumi_gcp          7.32.0
pulumi_kubernetes   4.15.0
python-dotenv       1.0.1
setuptools          70.1.1
supertokens-python  0.23.1
uvloop              0.19.0
virtualenv          20.26.3
watchfiles          0.22.0
websockets          12.0
wheel               0.43.0

Pulumi locates its logs in /tmp by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[guineveresaenger]

Hi @ocervell - thank you for reporting this issue, and we're sorry you're having trouble.

I've been able to reproduce this behavior. Here's what happens when I look at the details after pulumi preview:

Do you want to perform this update? details
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev::gcp-2247::pulumi:pulumi:Stack::gcp-2247-dev]
    ++gcp:container/cluster:Cluster: (create-replacement)
        [id=projects/pulumi-development/locations/europe-west1/clusters/gke-cluster-417a320]
        [urn=urn:pulumi:dev::gcp-2247::gcp:container/cluster:Cluster::gke-cluster]
        [provider=urn:pulumi:dev::gcp-2247::pulumi:providers:gcp::default_7_32_0::0c4bfd73-9d2d-4a0b-a09e-90d52d9b3e77]
      ~ nodeConfig: {
          ~ oauthScopes        : [
              ~ [0]: "https://www.googleapis.com/auth/monitoring" => "https://www.googleapis.com/auth/cloud-platform"
              - [1]: "https://www.googleapis.com/auth/devstorage.read_only"
              - [2]: "https://www.googleapis.com/auth/logging.write"
              - [3]: "https://www.googleapis.com/auth/service.management.readonly"
              - [4]: "https://www.googleapis.com/auth/servicecontrol"
              - [5]: "https://www.googleapis.com/auth/trace.append"
            ]
          - reservationAffinity: {
              - consumeReservationType: "NO_RESERVATION"
              - key                   : ""
              - values                : []
            }
          - reservationAffinity: {
              - consumeReservationType: "NO_RESERVATION"
              - key                   : ""
              - values                : []
            }
        }
    +-gcp:container/cluster:Cluster: (replace)
        [id=projects/pulumi-development/locations/europe-west1/clusters/gke-cluster-417a320]
        [urn=urn:pulumi:dev::gcp-2247::gcp:container/cluster:Cluster::gke-cluster]
        [provider=urn:pulumi:dev::gcp-2247::pulumi:providers:gcp::default_7_32_0::0c4bfd73-9d2d-4a0b-a09e-90d52d9b3e77]
      ~ nodeConfig: {
          ~ oauthScopes        : [
              ~ [0]: "https://www.googleapis.com/auth/monitoring" => "https://www.googleapis.com/auth/cloud-platform"
              - [1]: "https://www.googleapis.com/auth/devstorage.read_only"
              - [2]: "https://www.googleapis.com/auth/logging.write"
              - [3]: "https://www.googleapis.com/auth/service.management.readonly"
              - [4]: "https://www.googleapis.com/auth/servicecontrol"
              - [5]: "https://www.googleapis.com/auth/trace.append"
            ]
          - reservationAffinity: {
              - consumeReservationType: "NO_RESERVATION"
              - key                   : ""
              - values                : []
            }
          - reservationAffinity: {
              - consumeReservationType: "NO_RESERVATION"
              - key                   : ""
              - values                : []
            }
        }
    --gcp:container/cluster:Cluster: (delete-replaced)
        [id=projects/pulumi-development/locations/europe-west1/clusters/gke-cluster-417a320]
        [urn=urn:pulumi:dev::gcp-2247::gcp:container/cluster:Cluster::gke-cluster]
        [provider=urn:pulumi:dev::gcp-2247::pulumi:providers:gcp::default_7_32_0::0c4bfd73-9d2d-4a0b-a09e-90d52d9b3e77]

This unfortunately appears to be a limitation of GKE autopilot in conjunction with node_config.oauth_scopes that's been the case for quite a while. The recommendation seems to be to not use node_config.oauth_scopes.

It looks like you're running into the same issue as this Terraform user: https://github.com/hashicorp/terraform-provider-google/issues/13542.