kops 1.26.2 no longer applying additionalPolicies.master policy

[jim-barber-he]

/kind bug

1. What kops version are you running? The command kops version, will display this information.

$ kops version
Client version: 1.26.2 (git-v1.26.2)

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

 $ kubectl version    
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.2", GitCommit:"fc04e732bb3e7198d2fa44efa5457c7c6f8c0f5b", GitTreeState:"clean", BuildDate:"2023-02-22T13:39:03Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6", GitCommit:"ff2c119726cc1f8926fb0585c74b25921e866a28", GitTreeState:"clean", BuildDate:"2023-01-18T19:15:26Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

I have a 1.25.6 cluster created with kops 1.25.3 with the following in the cluster spec

  additionalPolicies:
    master: |
      [
        {
          "Action": [
            "sts:AssumeRole"
          ],
          "Effect": "Allow",
          "Resource": [
            "arn:aws:iam::REDACTED:role/kiam.*"
          ]
        }
      ]

I am using kops 1.26.2 to update the cluster like so: kops update cluster

5. What happened after the commands executed?

The changes shown by kops contained:

  IAMRolePolicy/additional.masters.$CLUSTER_NAME
        PolicyDocument
                                - {
                                -   "Statement": [
                                -     {
                                -       "Action": [
                                -         "sts:AssumeRole"
                                -       ],
                                -       "Effect": "Allow",
                                -       "Resource": [
                                -         "arn:aws:iam::REDACTED:role/kiam.*"
                                -       ]
                                -     }
                                -   ],
                                -   "Version": "2012-10-17"
                                - }

So it is going to attempt to remove the policy from the control plane nodes which is not what I want it to do.

6. What did you expect to happen?

The additional policy for the control plane nodes should stay intact.

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

Probably not applicable as I've given the spec.additionalPolicies.master part above that is no longer being honoured by kops.

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

Probably not applicable, but can send if requested. I ran it and caught stderr to a file and the output from spec_builder.go contains this:

  "additionalPolicies": {
    "control-plane": "[\n  {\n    \"Action\": [\n      \"sts:AssumeRole\"\n    ],\n    \"Effect\": \"Allow\",\n    \"Resource\": [\n      \"arn:aws:iam::REDACTED:role/kiam.*\"\n    ]\n  }\n]\n"
  },

It also has a line like:

I0315 15:41:32.768334  205190 topological_sort.go:81]   IAMRolePolicy/additional.masters.$CLUSTER_NAME:  [IAMRole/masters.$CLUSTER_NAME]

9. Anything else do we need to know?

I also tried changing:

  additionalPolicies:
    master: |

to

  additionalPolicies:
    control-plane: |

This is accepted but ends up getting turned back into spec.additionalPolicies.master again in the final cluster spec and still a kops update cluster results in the extra policy going to be removed.

I also stood up a fresh 1.26.2 cluster with kops 1.26.2 with a manifest setting the additional policy and the resulting IAM policy does not contain it.

[DerrickMartinez]

When editing via edit cluster, putting control-plane instead of master validates, but is not applied

[gregkoganvmm]

FYI, as a workaround you can add an in-line policy to the masters.your-cluster-name IAM policy - after it is manually added cluster edit/update does not pick it up for removal, so it kinda works (until the policy needs to be recreated from config of course). Hopefully the new release can be available soon to fix this.