[kops 1.8.0 AWS] Voyager 6.0.0-rc.0 HostPort ingress fails to successfully update security group

[annismckenzie]

I used Kops 1.8.0 to provision a new cluster on AWS. When trying to create a HostPort ingress I get the error:

Warning  FirewallUpdateFailed           8s  voyager operator  Failed to reconcile firewall. Reason: error authorizing security group ingress: UnauthorizedOperation: You are not authorized to perform this operation.

After decoding the authorization message failure via AWS STS I see that kops adds a StringEquals condition to the IAM policy for the master node – the code for that is here: https://github.com/kubernetes/kops/blame/15c7d61dfbe7ea86e97e6f32f137fd5dd388a91d/pkg/model/iam/iam_builder.go#L588.

After manually adding the KubernetesCluster tag to the security group Voyager creates it successfully updates the security group and opens the host port.

I tracked it down (hopefully to the right place) to $GOPATH/src/github.com/appscode/voyager/third_party/forked/cloudprovider/providers/aws/aws.go line 990 where I think adding the KubernetesCluster tag (which is already available in that file) would fix this? I'm currently in the process of trying to build our own Voyager images so I can test it but still wanted to report it. I'm also experiencing #889 in that same cluster.

We can chat on Slack if you need me to test something or provide more information. Thank you for Voyager, it's been great so far! :)

[r-divakaran-hrs]

Today I tried to create a NodePort ingress and stumbled on same error. The cluster was created using Kops version 1.8.3, but recently cluster is upgraded to version 1.9.3 using kops version 1.9.0. The voyager version am using is 6.0.0

[tamalsaha]

Yeah. This is an important issue. I am going to take a look. The reason I am procrastinating is because I have to setup a kops cluster and test things.

[annismckenzie]

To get it working right now, you only need to add the KubernetesCluster tag to the security group that already exists.

[r-divakaran-hrs]

I checked the security group created by voyager in my cluster and it already has that tag, but the error remains.

[annismckenzie]

Did you check whether the IAM permissions are correctly set? Depending on how you configured it you might need to set additionalPolicies for your cluster in the cluster spec like so:

additionalPolicies:
    node: |
      [
        {
          "Effect": "Allow",
          "Action": [
            "ec2:AuthorizeSecurityGroupIngress",
            "ec2:CreateRoute",
            "ec2:CreateSecurityGroup",
            "ec2:CreateTags",
            "ec2:DeleteRoute",
            "ec2:DeleteSecurityGroup",
            "ec2:ModifyInstanceAttribute",
            "ec2:RevokeSecurityGroupIngress",
            "ec2:DescribeInstances",
            "ec2:DescribeRouteTables",
            "ec2:DescribeSecurityGroups",
            "ec2:DescribeSubnets"
          ],
          "Resource": "*"
        }
      ]

[r-divakaran-hrs]

Yes I enabled extra policies and was able to over come that error. :) Thanks