No permissions to access to underlying AWS EKS Cluster directly

[wombelix]

Hi,

after finishing the Rancher Setup, there is this message:

In case you need to manage the Rancher cluster, you can generate your kubeconfig file using the command:
aws eks update-kubeconfig --region us-east-1 --name suse-rancher-setup-1961-cluster
For more details see the AWS documentation.

I can download the kubeconfig file for the cluster with the above command, but that doesn't mean that my user has access:

(⎈|suse-rancher-setup-1961-cluster:default)  ➜  tmp kubectl get nodes                                           
I0426 16:35:18.046381   48627 versioner.go:58] the server has asked for the client to provide credentials
E0426 16:35:19.609554   48627 memcache.go:238] couldn't get current server API group list: the server has asked for the client to provide credentials
E0426 16:35:20.223192   48627 memcache.go:238] couldn't get current server API group list: the server has asked for the client to provide credentials
E0426 16:35:20.833291   48627 memcache.go:238] couldn't get current server API group list: the server has asked for the client to provide credentials
E0426 16:35:21.430669   48627 memcache.go:238] couldn't get current server API group list: the server has asked for the client to provide credentials
E0426 16:35:22.046671   48627 memcache.go:238] couldn't get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server (the server has asked for the client to provide credentials)

The problem is, only the user or assumed Role that created the EKS Cluster has system:masters access permissions by default. In case of the Rancher Setup, that's the IAM Role / Instance Profile I created prior the setup and attached to the EC2 instance. Without an additional step, access to the EKS Cluster is only possible through Rancher but not directly.

In my case I lost connection to Rancher as I tried to change the TLS certificate from self-signed to Let's Encrypt and already deleted the rancher setup instance. I needed direct access to the EKS control plane to fix the issue, so my "solution" was to spin up a new EC2 instance, attach the IAM Instance Profile, install aws-cli and kubectl and then to add the role I needed to the aws-auth configmap manually:

aws eks update-kubeconfig --region us-east-1 --name suse-rancher-setup-1961-cluster
kubectl edit configmap aws-auth --namespace kube-system

[…]
    - rolearn: arn:aws:iam::1234567890:role/MyRole
      username: admin
      groups:
        - system:masters
[…]

My understand is that, if I had also already deleted the IAM instance profile, I've would be completely locked out from the EKS control plane with probably no chance to recover other than deleting and re-creating the whole Cluster.

Rancher setup should provide a mechanism to ensure that another user / role - other than the instance profile - has system:masters permission on the EKS Cluster. This could for example be achieved by adding another setup step to ask for the user / role ARN that should be granted full admin permissions.

Dom

[kevinayres]

This PR is from Amazon re: customers wanting to access the underlying cluster. Can we move to a service role approach via IRSA with the next release/product? Instance profiles are a deprecated approach now wherever disparate workloads exist such as k8s. If it's a single server function per node (like SUMA) then instance profiles are still appropriate.

[bear454]

The permissions requested for the Rancher Setup IAM role are not specific to the created cluster (as it obviously doesn't exist when assigned). There's no reason the same role or permissions cannot be delegated to another user, but that seems far outside the scope of Rancher Setup.

[rjschwei]

Given that we have a demonstrated use case where a user needed to access the cluster and needed to take extra steps to gain such access as the downloaded config did not work I am wondering how difficult it would be to add a step in the UI where the user provides an ARN that we can then configure as system::master. While this can certainly be considered outside the rancher-setup scope it is a user convenience feature that will help with day 2 operations.

[wombelix]

The permissions requested for the Rancher Setup IAM role are not specific to the created cluster

I'm not sure I understand your point, to use Rancher Setup it's required to create an appropriate IAM Role and attach it to the EC2 Instance. This role becomes then the only owner of the new created EKS Cluster.

A guided setup wizard that takes away the heavy lifting from the Customer shouldn't leave them alone when it comes to accessing the Cluster in my opinion.

Also the provided command at the end of Rancher Setup just doesn't work without additional steps as of today.

In case you need to manage the Rancher cluster, you can generate your kubeconfig file using the command:
aws eks update-kubeconfig --region us-east-1 --name suse-rancher-setup-1961-cluster
For more details see the AWS documentation.

So in my humble opinion, it is the responsibility of Rancher Setup to either provide clear guidance about the behaviour and what a Customer is supposed to do as post-setup step, or to handle it for the customer as part of the Setup.

as the downloaded config did not work I am wondering how difficult it would be to add a step in the UI where the user provides an ARN that we can then configure as system::master

That would probably to most customer friendly solution to this problem. A page to optional add a user or role ARN with a clear information why that's recommend and what need to be done afterwards if the step is skipped.

While this can certainly be considered outside the rancher-setup scope

As stated above, if a guided wizard is provided to take away complexity and to perform the whole deployment, I have the impression that such a topic start to fall into the scope of the setup to avoid getting locked out of the own Cluster.

My Ruby skills are not super strong, but if someone points me to the relevant files to understand better how the frontend and backend logic can be added, I'm happy to give it a shot and to contribute to a solution.

[kevinayres]

@rjschwei @bear454 will this modification make it into the next Marketplace listing? I think that many consumers assume there will be direct cluster access without additional steps.

[bear454]

Obviously it hasn't yet, due to other development priorities.