EKS cluster parameter `bootstrapClusterCreatorAdminPermissions` has wrong default

flostadler commented 4 months ago

What happened?

The documentation mentions that bootstrapClusterCreatorAdminPermissions of EKS clusters defaults to true. This is also the default behavior for the case when no ClusterAccessConfig is defined.

But in reality the bootstrapClusterCreatorAdminPermissions defaults to false.

This can cause the replacement of the whole cluster because changes to this parameter trigger replacements.

Example

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

const vpc = new awsx.ec2.Vpc("vpc", {
    subnetSpecs: [
        { type: "Public" }
    ],
    natGateways: {
        strategy: "None",
    }
});

const role = new aws.iam.Role("eksRole", {
    assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({ Service: "eks.amazonaws.com" }),
});

// Attach the AmazonEKSClusterPolicy to the role
new aws.iam.RolePolicyAttachment("eksClusterPolicy", {
    role: role.name,
    policyArn: "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy",
});

const cluster = new aws.eks.Cluster("my-cluster", {
    roleArn: role.arn,
    version: "1.29",
    vpcConfig: {
        subnetIds: vpc.publicSubnetIds,
    },
    accessConfig: {
        authenticationMode: "API_AND_CONFIG_MAP",
        // bootstrapClusterCreatorAdminPermissions: true, <-- should default to true if not specified, but defaults to false
    }
});

After deploying the snippet above, observe the cluster being created without admin permissions for the IAM principal that created the cluster.

This can also be seen in the CloudTrail event of the cluster creation:

{
...
    "accessConfig": {
        "authenticationMode": "API_AND_CONFIG_MAP",
        "bootstrapClusterCreatorAdminPermissions": false
    },
...
}

Output of `pulumi about`

CLI
Version      3.116.0
Go Version   go1.22.2
Go Compiler  gc

Plugins
KIND      NAME    VERSION
resource  aws     6.37.1
resource  awsx    2.10.0
resource  docker  4.5.4
resource  docker  3.6.1
language  nodejs  unknown

Host
OS       darwin
Version  14.4
Arch     arm64

This project is written in nodejs: executable='/opt/homebrew/bin/node' version='v22.1.0'

Current Stack: florian-pulumi-corp/eks-wrong-default/flo-eks-wrong-default

TYPE                                                 URN
pulumi:pulumi:Stack                                  urn:pulumi:flo-eks-wrong-default::eks-wrong-default::pulumi:pulumi:Stack::eks-wrong-default-flo-eks-wrong-default
pulumi:providers:awsx                                urn:pulumi:flo-eks-wrong-default::eks-wrong-default::pulumi:providers:awsx::default_2_10_0
pulumi:providers:aws                                 urn:pulumi:flo-eks-wrong-default::eks-wrong-default::pulumi:providers:aws::default_6_37_1
awsx:ec2:Vpc                                         urn:pulumi:flo-eks-wrong-default::eks-wrong-default::awsx:ec2:Vpc::vpc
aws:iam/role:Role                                    urn:pulumi:flo-eks-wrong-default::eks-wrong-default::aws:iam/role:Role::eksRole
pulumi:providers:aws                                 urn:pulumi:flo-eks-wrong-default::eks-wrong-default::pulumi:providers:aws::default_6_32_0
aws:iam/rolePolicyAttachment:RolePolicyAttachment    urn:pulumi:flo-eks-wrong-default::eks-wrong-default::aws:iam/rolePolicyAttachment:RolePolicyAttachment::eksClusterPolicy
aws:ec2/vpc:Vpc                                      urn:pulumi:flo-eks-wrong-default::eks-wrong-default::awsx:ec2:Vpc$aws:ec2/vpc:Vpc::vpc
aws:ec2/internetGateway:InternetGateway              urn:pulumi:flo-eks-wrong-default::eks-wrong-default::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/internetGateway:InternetGateway::vpc
aws:ec2/subnet:Subnet                                urn:pulumi:flo-eks-wrong-default::eks-wrong-default::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::vpc-public-2
aws:ec2/subnet:Subnet                                urn:pulumi:flo-eks-wrong-default::eks-wrong-default::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::vpc-public-1
aws:ec2/subnet:Subnet                                urn:pulumi:flo-eks-wrong-default::eks-wrong-default::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::vpc-public-3
aws:ec2/routeTable:RouteTable                        urn:pulumi:flo-eks-wrong-default::eks-wrong-default::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::vpc-public-2
aws:ec2/routeTable:RouteTable                        urn:pulumi:flo-eks-wrong-default::eks-wrong-default::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::vpc-public-1
aws:ec2/routeTable:RouteTable                        urn:pulumi:flo-eks-wrong-default::eks-wrong-default::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::vpc-public-3
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:flo-eks-wrong-default::eks-wrong-default::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::vpc-public-2
aws:ec2/route:Route                                  urn:pulumi:flo-eks-wrong-default::eks-wrong-default::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::vpc-public-2
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:flo-eks-wrong-default::eks-wrong-default::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::vpc-public-1
aws:ec2/routeTableAssociation:RouteTableAssociation  urn:pulumi:flo-eks-wrong-default::eks-wrong-default::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::vpc-public-3
aws:ec2/route:Route                                  urn:pulumi:flo-eks-wrong-default::eks-wrong-default::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::vpc-public-1
aws:ec2/route:Route                                  urn:pulumi:flo-eks-wrong-default::eks-wrong-default::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::vpc-public-3
pulumi:providers:pulumi                              urn:pulumi:flo-eks-wrong-default::eks-wrong-default::pulumi:providers:pulumi::default

Found no pending operations associated with flo-eks-wrong-default

Backend
Name           pulumi.com
URL            https://app.pulumi.com/florian-pulumi-corp
User           florian-pulumi-corp
Organizations  florian-pulumi-corp, pulumi
Token type     personal

Dependencies:
NAME            VERSION
@pulumi/pulumi  3.117.0
@types/node     18.19.33
typescript      5.4.5
@pulumi/aws     6.37.1
@pulumi/awsx    2.10.0

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

flostadler commented 4 months ago

This seems to be an upstream bug: https://github.com/hashicorp/terraform-provider-aws/issues/36259

flostadler commented 2 months ago

The root cause of this is how boolean values are handled in terraform-plugin-sdk: https://github.com/hashicorp/terraform-plugin-sdk/issues/817.

A boolean value being set to false and a boolean value being non-existent are handled the same way. This in combination with EKS defaulting the parameter to true leads to this problem.

We could fix the docs to say it's defaulting to false, but that's not entirely true either. If the accessConfig block is not specified, it's actually correctly defaulting to true because the nested bootstrapClusterCreatorAdminPermissions isn't even evaluated. The linked issue mentions that we might be able to work around the wrong defaulting by inspecting the raw configuration (by using GetRawConfig) but that isn't supported for computing changes.

flostadler commented 2 months ago

Well, the upstream bug (https://github.com/hashicorp/terraform-provider-aws/issues/36259) has been fixed yesterday evening. I'm gonna test if it solves the problems on the pulumi side as well, if it does even better because we don't need a patch.

flostadler commented 2 months ago

The upstream fix went with the option of having the bi-modal default behavior I described above. If accessConfig is not specified, bootstrapClusterCreatorAdminPermissions defaults to true. If accessConfig is provided but bootstrapClusterCreatorAdminPermissions is not set, it defaults to false.

That's not perfect, but after looking at the upstream code it's the closest we can get without introducing breaking changes. The problem with bootstrapClusterCreatorAdminPermissions is that it's not returned by the GetCluster API call and that it forces replacements. Forcing replacements for EKS clusters is extremely disruptive, because it deletes the clusters and all resources running on it.

We'll pull this fix in as part of our next upstream upgrade which will happen as soon as hashicorp/terraform-provider-aws cuts a new release.

pulumi-bot commented 2 months ago

This issue has been addressed in PR #4217 and shipped in release v6.45.0.

pulumi / pulumi-aws