search

Azure / AKS

Azure Kubernetes Service
https://azure.github.io/AKS/
1.96k stars 306 forks source link

RBAC AAD access error. You must be logged in to the server (Unauthorized). AKS 1.10.3 #478

Closed MarkTopping closed 5 years ago

MarkTopping commented 6 years ago

After significant hours invested in trying, I'm unable to access cluster resources under my AAD account (thus as a non-admin user) when RBAC is enabled.

I've followed & re-followed the steps to create a cluster with RBAC / AAD as found here: https://github.com/Azure/acs-engine/blob/master/docs/kubernetes/aad.md

The only different path I took is I wanted to create my cluster using the Resource Templates rather than via the Azure CLI. Thus I used the "2018-03-31" template and set enableRBAC=true and provided a AADProfile section nested within properties: {...}. My cluster was created successfully with the template - using Kubernetes version 1.10.3.

Now as it stands, when I connect to my cluster (as non-admin) then I am requested to authenticate at https://microsoft.com/devicelogin and upon doing so the website confirms I have authenticated with my AAD Client (as setup as a native App Registration in my Azure AD).

However, once my cli updates itself, I'm presented with the message: "You must be logged in to the server (Unauthorized)".

It might be of interest to note that while logged in as admin, if I try kubectl get pods --as=MyUserName then the command works. And If I run kubectl auth can-i get pods --as=myUserName then it responds with a 'yes'. These repsonses very much contradict what I witness when I actually try to interact with my cluster under my own credentials.

FYI I've tried created cluster role bindings for both AAD Groups and just for a single user with the same outcome. An example binding I've applied is:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: sorted-react-nonprod-admins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: "myUserName"

I've tried both username and e-mail in the name field. Along with AD Group names with 'kind' set to 'Group'

joakimhellum commented 6 years ago

Can successfully create Azure AD-enabled clusters by following these instructions carefully: https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/aks/aad-integration.md

However it's easy to make mistakes when configuring in the Azure portal, and for me some mistakes resulted in similar error message. For example:

MarkTopping commented 6 years ago

Hi @joakimhellum-in Thanks for your suggestions. I can confirm the following:

However, with respect to your last comment about a guest account... maybe there is something in that. When viewing my own profile in the AAD via the Portal, there is a greyed out box labelled 'User type' and within it it says 'Guest'.

Despite my Role as a Global Admin, are you suggesting that this guest user type might be the cause of this issue?

joakimhellum commented 6 years ago

Despite my Role as a Global Admin, are you suggesting that this guest user type might be the cause of this issue? Yes, I always get the error message "error: You must be logged in to the server (Unauthorized)" when attempting to login with Guest accounts, but using Member accounts works fine. I will open support ticket in Azure regarding this later today.

MarkTopping commented 6 years ago

@joakimhellum-in Ok, so to test the otherside of the water here; I've justed added someone who has a 'Member' account into my AD Administrator group; and that group has been given a 'cluser-admin' cluster role binding in my AKS cluster.

However, when I run the command: kubectl get pods --as=memberAccountNameHere I still get the the following:

Error from server (Forbidden): pods is forbidden: User "[name removed]" cannot list pods in the namespace "default"

and equally, running: kubectl auth can-i get pods --as=MemberAccountEmail returns no

I cannot simulate a happy path using a Member Account either I'm afraid

joakimhellum commented 6 years ago

@MarkTopping Great, this error message Error from server (Forbidden): pods is forbidden: User "[name removed]" cannot list pods in the namespace "default" is very familiar. To me this is normally just updating ClusterRoleBinding, for example:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: demo-cluster-admins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: "user@example.com"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: dashboard-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system

It's the config errors related to error: You must be logged in to the server (Unauthorized) that caused the most pain :)

To be clear after adding the ClusterRoleBinding I only run these commands:

az login -u user@example.com
az aks get-credentials -n <my-cluster> -g <my-resource-group>
kubectl get pods
MarkTopping commented 6 years ago

@joakimhellum-in Ok cool... your responses have got me somewhere :)

While my get resource as another user impersonation attempts seemed to suggest a Member account would not work....

I pursued and created a Member account for myself with the desired results. I'm both access the AKS resources now when both configured as a single user or within a group.

Thank you for your many responses ... very helpful!

You mentioned you were opening this as a support ticket.... I assume that is it say that you think not being able to use 'Guest' accounts is a bug (or oversight) that will need to be fixed in the not to distant future?

(I don't fancy having to create new Azure accounts for the entire organisation :-( )

joakimhellum commented 6 years ago

@MarkTopping Yes, we have opened support ticket in Azure regarding using Guest accounts, don't know if it's supported or not at this time. Will update this comment later.

vishnoisuresh commented 6 years ago

We are having the same situations as our Azure AD accounts are type guest due to Active Directory federations (with on-premise AD). We can not create accounts in Azure Active Directory because we need 2-factors auths.

This is a blocker as we could not move to production.

shahiddev commented 6 years ago

Just spent too long battling with similar issues - initially not pressing the grant permissions (RTFM!) and then the account I was using was a Guest from a different AD!

strtdusty commented 6 years ago

One thing we discovered with Microsofts help today is that you must have the Microsoft Graph API selected when you do the grant permissions step.

amanohar commented 6 years ago

@vishnoisuresh @shahiddev some cases where AAD application is in a different tenant than the user is not supported at this time (this is an example of B2B collaboration: https://docs.microsoft.com/en-us/azure/active-directory/b2b/claims-mapping). Support will be added for this at some point.

MohamedKhelifi commented 6 years ago

Hello guys,

I followed this documentation https://docs.microsoft.com/en-us/azure/aks/aad-integration and the "Grant permission" step was done by a tenant admin but I have the same issue. Note that my account type is "Member" (not a Guest) and the AAD application is in a same tenant as the users :

:~/$ kubectl get node --v=9
I1023 15:55:20.940020    6015 loader.go:359] Config loaded from file /home/mohamed/.kube/config
I1023 15:55:20.942535    6015 loader.go:359] Config loaded from file /home/mohamed/.kube/config
I1023 15:55:20.973925    6015 loader.go:359] Config loaded from file /home/mohamed/.kube/config
I1023 15:55:20.982001    6015 loader.go:359] Config loaded from file /home/mohamed/.kube/config
I1023 15:55:20.982535    6015 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json;as=Table;v=v1beta1;g=meta.k8s.io, application/json" -H "User-Agent: kubectl/v1.12.1 (linux/amd64) kubernetes/4ed3216" 'https://blablablablaa-03d4471b.hcp.westeurope.azmk8s.io:443/api/v1/nodes?limit=500'
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code CCK58D9NJ to authenticate.
I1023 15:55:42.760752    6015 loader.go:359] Config loaded from file /home/mohamed/.kube/config
I1023 15:55:42.767321    6015 loader.go:359] Config loaded from file /home/mohamed/.kube/config
I1023 15:55:42.793539    6015 loader.go:359] Config loaded from file /home/mohamed/.kube/config
I1023 15:55:43.393762    6015 round_trippers.go:405] GET https://blblaablabla-03d4471b.hcp.westeurope.azmk8s.io:443/api/v1/nodes?limit=500 401 Unauthorized in 22411 milliseconds
I1023 15:55:43.393852    6015 round_trippers.go:411] Response Headers:
I1023 15:55:43.393876    6015 round_trippers.go:414]     Content-Type: application/json
I1023 15:55:43.393897    6015 round_trippers.go:414]     Content-Length: 129
I1023 15:55:43.393914    6015 round_trippers.go:414]     Date: Tue, 23 Oct 2018 13:55:43 GMT
I1023 15:55:43.393997    6015 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I1023 15:55:43.394999    6015 helpers.go:201] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}]
F1023 15:55:43.395183    6015 helpers.go:119] error: You must be logged in to the server (Unauthorized)

My yaml file:

:~/Documents/GIT/Infrastructure_Deployment_on_Azure/Test/kubernetes/rbac$ cat rbac_admin.yaml 
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: nit-admin-cluster
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: "mohamed.khelifi@toto.onmicrosoft.com"
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: "Mohamed.KHELIFI@toto.com"
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: "mohamed.khelifi@toto.com"

Any help will be appreciated.

Thanks.

amanohar commented 5 years ago

@MohamedKhelifi did the tenant admin use Portal to Grant Permissions? If yes, can I request to you ask them again to perform step 7 here: https://docs.microsoft.com/en-us/azure/aks/aad-integration#create-server-application and step 4 here: https://docs.microsoft.com/en-us/azure/aks/aad-integration#create-server-application again? Make sure "Microsoft Graph" is select in step 7 before clicking grant permissions.

Also, can you share your resource group name and resource name + approx. time you tried this operation that resulted in Unauthorized?

MohamedKhelifi commented 5 years ago

@amanohar Thanks for your reply, Yes, the tenant admin use Portal to Grant Permissions. I can ask them tomorrow morning but they did it with me, I'm sure about this step. What do you mean about : Make sure "Microsoft Graph" is highlighted in step 7 ?

I have this in Azure portal: image And here we can see the permissions granted by tenant admin: image My resource group name and resource name : 

az aks get-credentials --resource-group RgTIcoma01 --name KsTIcomA01

When I run "kubectl get nodes or anything , I'm redirect to /devicelogin and I have to connect with my user account AD after this I have the error : error: You must be logged in to the server (Unauthorized) This is take few seconds...

Thanks.

amanohar commented 5 years ago

@MohamedKhelifi I searched the logs and see this: "error_description\":\"AADSTS70002: Error validating credentials. AADSTS50012: Invalid client secret is provided. It looks like you provided invalid secret for your AAD app or app secret was invalidated.

MohamedKhelifi commented 5 years ago

@amanohar Thanks for your help, I will check the client secret. Can I ask you how did you see this logs? Thank you very much.

amanohar commented 5 years ago

@MohamedKhelifi steps to enable diagnostic logs for your control plane here: https://docs.microsoft.com/en-us/azure/aks/view-master-logs

To search for logs related to your AAD authorization run the following query once you have followed steps mentioned in the link above:

search *
| where Type == "AzureDiagnostics"
| where Category == "guard"
| project log_s
MohamedKhelifi commented 5 years ago

@amanohar Thanks for your help, it works now. You're right the client secret was wrong. Thank you very much.