Closed bergerx closed 5 years ago
Dex as federated OpenID Connect provider cannot work because it also must be configured at the api server in order to work. Since you don't have access to the aks control plane it could never work.
You will have to use the ADD/OIDC server connected to your apiserver directly (without dex) in your dashboard configuration. The tennant_id
ist the Directory-ID you will find in the Azure Active Directory > Properties. As application_id
I would try to use the Server application ID
since it is of type WebApp. In the manifest of this you will have to enable implicit outh flow "oauth2AllowImplicitFlow": true
and you have add the dashboard-callback-url
as redirect uri to the Server Application
https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-configure-app-access-web-apis#add-redirect-uris-to-your-application. Unfortunately I am not able to try that this works with our ADD :-(
jwt:
audience: "{{application_id}}"
issuer: "https://sts.windows.net/{{tenant_id}}/"
algorithms: [ RS256 ]
jwks:
strictSsl: true
rejectUnauthorized: true
cache: true
rateLimit: false
jwksRequestsPerMinute: 5
jwksUri: https://login.microsoftonline.com/common/discovery/keys
frontend:
oidc:
authority: "https://sts.windows.net/{{tenant_id}}/"
client_id: "{{application_id}}"
redirect_uri: "{{public_dashboard_url}}/callback"
response_type: 'token id_token'
scope: 'openid email profile groups'
loadUserInfo: false
I hope this hints will help you.
Best regards, Holger
Hey, thanks a lot for the pointers, previously we tried to run this with @praveendhac.
I'm not sure about the impact of changing the "oauth2AllowImplicitFlow": true
and dashboard-callback-url
in the AKS's integration since it requires certain configuration.
A recent trial has been done here: https://kubernetes.slack.com/archives/CB57N0BFG/p1544718587089700
I am stuck with error, AADSTS50001: Resource identifier is not provided. Using below command to deploy chart
$ helm upgrade --install --namespace kube-system --values charts/gardener-dashboard/pd-values.yaml gardener-dashboard charts/gardener-dashboard
gardener-dashboard values.yaml
$ cat ../pd-values.yaml
# Default values for gardener-dashboard.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
image:
repository: eu.gcr.io/gardener-project/gardener/dashboard
tag: latest
pullPolicy: Always
logLevel: debug
apiServerUrl: https://my-aks-k8s-apiserver.hcp.eastus.azmk8s.io
containerPort: 8080
servicePort: 8080
resources:
limits:
cpu: 250m
memory: 256Mi
requests:
cpu: 100m
memory: 128Mi
hosts:
- pd-dshboard.ingress.msa-sandbox.example.com
tls:
crt: |
-----BEGIN CERTIFICATE-----
Li4u
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
Li4u
-----END CERTIFICATE-----
key: |
-----BEGIN RSA PRIVATE KEY-----
Li4u
-----END RSA PRIVATE KEY-----
jwt:
audience: "azure-appreg-appid"
issuer: https://sts.windows.net/azure-my-tenant-id/
algorithms: [ RS256 ]
jwks:
strictSsl: true
rejectUnauthorized: true
cache: true
rateLimit: false
jwksRequestsPerMinute: 5
jwksUri: https://login.microsoftonline.com/common/discovery/keys
frontend:
oidc:
authority: "https://sts.windows.net/azure-my-tenant-id/"
client_id: "azure-appreg-appid"
redirect_uri: "https://pd-dshboard.ingress.msa-sandbox.example.com/callback"
#redirect_uri: "https://sts.windows.net/azure-my-tenant-id/callback"
response_type: 'token id_token'
scope: 'openid email profile groups access_token'
loadUserInfo: false
oidc:
audience: "azure-appreg-appid"
issuer: https://sts.windows.net/azure-my-tenant-id/
issuerUrl: https://sts.windows.net/azure-my-tenant-id/
clientId: "azure-appreg-appid"
frontendConfig:
landingPageUrl: https://github.com/gardener
helpMenuItems:
- title: Getting Started
icon: description
url: https://github.com/gardener/gardener/tree/master/docs
- title: Issues
icon: bug_report
url: https://github.com/gardener/gardener/issues
gitHubRepoUrl: https://foo-github.com/dummyorg/dummyrepo
prometheus:
secret: secret
livenessProbe:
enabled: true
initialDelaySeconds: 15
periodSeconds: 20
timeoutSeconds: 5
failureThreshold: 6
successThreshold: 1
readinessProbe:
enabled: true
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 6
successThreshold: 1
Azure side (portal) config Reply URLs: https://pd-dshboard.ingress.msa-sandbox.example.com/callback App id manifest
"groupMembershipClaims": "All",
"oauth2AllowImplicitFlow": true,
Required permissions gave Graph, and AD permissions to "Read directory data" and "Sign in and read user profile"
It seems that Azure requires a resource to be specified when requesting an access_token.
https://social.msdn.microsoft.com/Forums/sqlserver/en-US/d4c7614e-30bd-4f78-83bb-63c3ce45bd0f/authentication-problem-with-microsoft-azure-application?forum=WindowsAzureAD
I would try to change the response_type in the configMap from token id_token
to id_token
.
https://github.com/gardener/dashboard/blob/master/charts/gardener-dashboard/templates/configmap.yaml#L57
Unfortunately I am not able to try Azure AD myself. So please excuse me if I am not able to really help.
According to the following documentation https://docs.microsoft.com/de-de/azure/active-directory/develop/v1-protocols-openid-connect-code#send-the-sign-in-request it should not be necessary to specify a resource if the the response_type is only id_token. Currently it is not possible to change the response_type via Helm values.yaml
Using v2.0 worked for below scenario https://stackoverflow.com/questions/43764585/resource-parameter-when-requesting-access-token
I tried v2.0 but getting Same Origin Policy errors.
We don't pass azure app-registration client secret, we only pass clientid, how is gardener-dashboard going to authenticate with AzureAD.
@praveendhac have you tried user response_type id_token
only and not token id_token
. This should solve the problem with AADSTS50001: Resource identifier is not provided. We do not use or authenticate with AzureAD since we cannot configure it for our needs.
Tried id_token, got same error "AADSTS50001: Resource identifier is not provided."
@holgerkoser gardener-dashboard AuthN flow is working but don't see anything on the UI. This is the change I made to configmap.yaml
- redirect_uri: "{{ .Values.oidc.issuerUrl }}/callback"
- response_type: "token id_token"
+ redirect_uri: "{{ .Values.frontend.oidc.redirect_uri }}"
+ response_type: "id_token"
redirect_uri is populating wrong values in configmap.yaml from values.yaml
frontend:
oidc:
redirect_uri: https://pd-dshboard.ingress.msa-sandbox.example.com/callback
<REDACTED>
oidc:
issuerUrl: https://sts.windows.net/<tenant_id>/
@praveendhac If you don't see anything in the UI open the developer console. Do you see any errors there e.g. problems with CORS....(if yes which errors do you see). I would also recommend to clear the browser cache.
Yeah, I am getting CSP error (csp:blocked) while accessing https://login.microsoftonline.com/common/discovery/keys
I prepared a PR #268 which should fix the problem that access to the jwksUrl is blocked by CSP rules. It also allows to configure frontend oidc-client via helm.
After the recent problem addressed, we hit several other issues, here are some issues i worked around to see if how far can I go. Apparently even if we solve the "Content Security Policy" and "Access-Control-Allow-Origin" problems seems like my user still seems to be not properly authenticated or either authorizes. Below you can find the steps we took to investigate.
After clicking the login link in the dashboard we saw this error in the browser console without any logs in the gardener-dashboard pod:
Refused to connect to 'https://login.microsoftonline.com/common/discovery/keys' because it violates the following Content Security Policy directive: "connect-src 'self' wss: ws: https://sts.windows.net/{{cropped}}/".
And here is a screenshot:
For now i'm able to work around the problem by installing Content-Security-Policy chrome extension and disabling the CSP at browser level. I'm not sure where to address this issue.
After working around the CSP issue and tried to login to the dashboard again this we got this one, and again without any logs appearing in the gardener-dashboard pod:
Failed to load https://login.microsoftonline.com/common/discovery/keys: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://pd-dshboard.ingress.{{cropped}}.io' is therefore not allowed access.
16:30:09.985
...
Cross-Origin Read Blocking (CORB) blocked cross-origin response https://login.microsoftonline.com/common/discovery/keys with MIME type application/json. See https://www.chromestatus.com/feature/5629709824032768 for more details.
And here is a screenshot:
For now, I'm able to work around the problem by installing chrome extension and disabling the CORS at the browser level. I'm not sure where to address this issue.
After disabling both CSP and CORS on my browser I was able to pass the login sequence but calls to the gardener-dashboard's /api endpoints were getting 401
2018-12-17T16:35:50.114Z - error: Error with invalid code credentials_bad_scheme: Format is Authorization: Bearer [token] UnauthorizedError: Format is Authorization: Bearer [token]
at middleware (/usr/src/app/node_modules/express-jwt/lib/index.js:64:25)
at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:317:13)
at /usr/src/app/node_modules/express/lib/router/index.js:284:7
at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:335:12)
at next (/usr/src/app/node_modules/express/lib/router/index.js:275:10)
at cors (/usr/src/app/node_modules/cors/lib/index.js:188:7)
at /usr/src/app/node_modules/cors/lib/index.js:224:17
at originCallback (/usr/src/app/node_modules/cors/lib/index.js:214:15)
at /usr/src/app/node_modules/cors/lib/index.js:219:13
2018-12-17T16:35:50.115Z - http:10.240.0.4 - - [17/Dec/2018:16:35:50 +0000] "GET /api/user HTTP/1.1" 401 135
2018-12-17T16:35:50.226Z - error: Error with invalid code credentials_bad_scheme: Format is Authorization: Bearer [token] UnauthorizedError: Format is Authorization: Bearer [token]
at middleware (/usr/src/app/node_modules/express-jwt/lib/index.js:64:25)
at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:317:13)
at /usr/src/app/node_modules/express/lib/router/index.js:284:7
at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:335:12)
at next (/usr/src/app/node_modules/express/lib/router/index.js:275:10)
at cors (/usr/src/app/node_modules/cors/lib/index.js:188:7)
at /usr/src/app/node_modules/cors/lib/index.js:224:17
at originCallback (/usr/src/app/node_modules/cors/lib/index.js:214:15)
at /usr/src/app/node_modules/cors/lib/index.js:219:13
2018-12-17T16:35:50.226Z - http:10.240.0.4 - - [17/Dec/2018:16:35:50 +0000] "GET /api/cloudprofiles HTTP/1.1" 401 135
2018-12-17T16:35:50.227Z - error: Error with invalid code credentials_bad_scheme: Format is Authorization: Bearer [token] UnauthorizedError: Format is Authorization: Bearer [token]
at middleware (/usr/src/app/node_modules/express-jwt/lib/index.js:64:25)
at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:317:13)
at /usr/src/app/node_modules/express/lib/router/index.js:284:7
at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:335:12)
at next (/usr/src/app/node_modules/express/lib/router/index.js:275:10)
at cors (/usr/src/app/node_modules/cors/lib/index.js:188:7)
at /usr/src/app/node_modules/cors/lib/index.js:224:17
at originCallback (/usr/src/app/node_modules/cors/lib/index.js:214:15)
at /usr/src/app/node_modules/cors/lib/index.js:219:13
2018-12-17T16:35:50.228Z - http:10.240.0.4 - - [17/Dec/2018:16:35:50 +0000] "GET /api/namespaces HTTP/1.1" 401 135
2018-12-17T16:35:50.229Z - error: Error with invalid code credentials_bad_scheme: Format is Authorization: Bearer [token] UnauthorizedError: Format is Authorization: Bearer [token]
at middleware (/usr/src/app/node_modules/express-jwt/lib/index.js:64:25)
at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:317:13)
at /usr/src/app/node_modules/express/lib/router/index.js:284:7
at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:335:12)
at next (/usr/src/app/node_modules/express/lib/router/index.js:275:10)
at cors (/usr/src/app/node_modules/cors/lib/index.js:188:7)
at /usr/src/app/node_modules/cors/lib/index.js:224:17
at originCallback (/usr/src/app/node_modules/cors/lib/index.js:214:15)
at /usr/src/app/node_modules/cors/lib/index.js:219:13
2018-12-17T16:35:50.229Z - http:10.240.0.4 - - [17/Dec/2018:16:35:50 +0000] "GET /api/domains HTTP/1.1" 401 135
2018-12-17T16:35:50.925Z - debug: Socket /shoots#L3-syp0iJZUKkwTTAAAR connected
2018-12-17T16:35:50.928Z - debug: Socket /journals#L3-syp0iJZUKkwTTAAAR connected
2018-12-17T16:35:51.027Z - debug: Socket /shoots#L3-syp0iJZUKkwTTAAAR authenticating
2018-12-17T16:35:51.030Z - debug: Socket /journals#L3-syp0iJZUKkwTTAAAR authenticating
2018-12-17T16:35:51.082Z - debug: Socket /shoots#L3-syp0iJZUKkwTTAAAR authenticated (user bdogan@xxx.com)
2018-12-17T16:35:51.092Z - debug: Socket /journals#L3-syp0iJZUKkwTTAAAR authenticated (user bdogan@xxx.com)
Checking the requests, i saw that the Authorisation requests are not right, the Authorisation: Bearer ..
header is passed as Authorisation: undefined ...
, here is an example:
TODO
As a workaround we tampered the connections in the browser and replaced undefined
with Bearer
and this time log in and after the gardener-apiserver requests are successful (200 OK with valid data returned):
Example reply from the https://pd-dshboard.ingress.{{cropped}}.io/api/cloudprofiles endpoint. so they work:
Here are the logs in gardener-dashboard during the login:
2018-12-17T19:59:29.373Z - debug: Socket /shoots#QCz6lMW0nBwClIvXAAAU disconnected. Reason: transport close
2018-12-17T19:59:29.373Z - debug: Socket /journals#QCz6lMW0nBwClIvXAAAU disconnected. Reason: transport close
2018-12-17T19:59:38.554Z - http:10.240.0.4 - - [17/Dec/2018:19:59:38 +0000] "GET /api/user HTTP/1.1" 200 42
2018-12-17T19:59:41.791Z - http:10.240.0.4 - - [17/Dec/2018:19:59:41 +0000] "GET /api/cloudprofiles HTTP/1.1" 200 1658
2018-12-17T19:59:42.853Z - http:10.240.0.4 - - [17/Dec/2018:19:59:42 +0000] "GET /api/namespaces HTTP/1.1" 200 2
2018-12-17T19:59:43.153Z - debug: Socket /shoots#DVHuMef10fH2xyRfAAAV connected
2018-12-17T19:59:43.157Z - debug: Socket /journals#DVHuMef10fH2xyRfAAAV connected
2018-12-17T19:59:43.261Z - debug: Socket /shoots#DVHuMef10fH2xyRfAAAV authenticating
2018-12-17T19:59:43.290Z - debug: Socket /shoots#DVHuMef10fH2xyRfAAAV authenticated (user bdogan@xxx.com)
2018-12-17T19:59:43.343Z - debug: Socket /journals#DVHuMef10fH2xyRfAAAV authenticating
2018-12-17T19:59:43.370Z - debug: Socket /journals#DVHuMef10fH2xyRfAAAV authenticated (user bdogan@xxx.com)
2018-12-17T19:59:45.091Z - http:10.240.0.4 - - [17/Dec/2018:19:59:45 +0000] "GET /api/domains HTTP/1.1" 200 127
But still the https://pd-dshboard.ingress.{{cropped}}.io/api/user endpoint returns no auth
{"isAdmin":false,"canCreateProject":false}
So we tried assigning RBAC rolebinding to the logged-in user:
[msa-sandbox-garden:garden]~ $ kubectl get clusterrolebindings -o wide | egrep 'bdogan|ROLE'
NAME AGE ROLE USERS GROUPS SERVICEACCOUNTS
bekir-can-cloudprofile 4h5m ClusterRole/garden.sapcloud.io:system:cloudprofiles bdogan@xxx.com
bekir-cluster-admin 5h44m ClusterRole/cluster-admin bdogan@xxx.com
bekir-gacan-create-project 4h31m ClusterRole/garden.sapcloud.io:system:project-creation bdogan@xxx.com
bekir-garden-admin 4h39m ClusterRole/garden.sapcloud.io:admin bdogan@xxx.com
bekir-garden-member 4h4m ClusterRole/garden.sapcloud.io:system:project-member bdogan@xxx.com
bekir-garden-system-admin 4h39m ClusterRole/garden.sapcloud.io:system:administrators bdogan@xxx.com
[msa-sandbox-garden:garden]~ $
But still no controls in the UI are enabled after logged in and we have no shoots/projects listed.
This should be solved with #268. Of course you do not see any logs in the dashboard pod because the it is not involved in this case. The browser directly talks to azure.
It seems the azure jwksUri endpoint does not support CORS. Only solution I see is to move to complete oidc flow to the backend and use authorization code flow and provide the token from the backend. Or ask Azure to support CORS for the keys endpoint.
This should be solved with #268.
I think the username in the token does not match the one confiurged in the bindings. How does the payload of the token look like? You can paste it to https://jwt.io/ for decoding.
1.27.0-dev-d827f3a
I have build an image which contains all changes of PR #268 and pushed it to gcr:
eu.gcr.io/gardener-project/gardener/dashboard:1.27.0-dev-d827f3a
@bergerx The problem with jwks endpoint is described here https://github.com/damienbod/angular-auth-oidc-client/issues/19
I think the username in the token does not match the one confiurged in the bindings. How does the payload of the token look like? You can paste it to https://jwt.io/ for decoding.
I checked the token, it seems to be matching my k8s username, im not sure if there are some other fields to check, here is how it looks like:
trying with the new image shortly
Just tried the new image, i can confirm that it solves 1 and 3. Thanks a lot for looking into this.
For 2:
Only solution I see is to move to complete oidc flow to the backend and use authorization code flow and provide the token from the backend. Or ask Azure to support CORS for the keys endpoint.
Can't we do both, first for short/mid term, second for long term
For 4:
The username in token and the rolbindings match. Do you have any other idea what could be other possible problems?
its bdogan@xxx.com
in the JWT token, see the screenshot above, also in gardener-dasboard logs i can see this line: 2018-12-18T10:49:25.762Z - debug: Socket /journals#OkIoQ6I2AJLTZ9FUAAAC authenticated (user bdogan@xxx.com)
and here is the rolebinding:
[msa-sandbox-garden:garden]~ $ kubectl get clusterrolebindings bekir-garden-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: 2018-12-17T15:26:29Z
name: bekir-garden-admin
resourceVersion: "18756087"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/bekir-garden-admin
uid: 208463c3-0210-11e9-a8be-0a58ac1f020a
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: garden.sapcloud.io:admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: bdogan@xxx.com
[msa-sandbox-garden:garden]~ $
Also have updated to the new image to check the connect-src issue. After updating the flowing CSP seems to be hit
Refused to frame 'https://identity.example.com/' because it violates the following Content Security Policy directive: "default-src 'self'". Note that 'frame-src' was not explicitly set, so 'default-src' is used as a fallback.
Because of this the following also appears.
Failed to execute 'postMessage' on 'DOMWindow': The target origin provided ('https://identity.example.com') does not match the recipient window's origin ('null').
@eaterm It will not work if you use the default helm value without any change. https://identity.example.com/
make no sense.
@holgerkoser I have changed the values in the helm chart. example.com is just to illustrate the issue. The errors i see have my domain in it.
@bergerx ad 2) Mid term we plan to move from implicit flow to authorization_code flow. But this is a larger change and we want to do this together with some other changes in the backend. But I have created a PR #270 which allows you to configure oidc metadata for the oidc-client. This should looks something like this:
oidc:
issuerUrl: &issuer https://sts.windows.net/{tenant}/
authority: https://login.microsoftonline.com/{tenant}/
...
metadata:
issuer: *issuer
authorization_endpoint: https://login.microsoftonline.com/{tenant}/oauth2/authorize
jwks_uri: /keys
If the value of oidc.metadata.jwks_uri === '/keys'
we proxy the jwks endpoint via our backend. I hope this solves your problem short term.
ad 4) What result do you get if you add a user with the token from the oidc to your kubeconfig and do the following:
kubectl auth can-i create projects
What do you find in the log of the api-server, why the token is not allowed to create projects? What ist the oidc configuration of your api-server?
I have create a dev image for the latest commit on the cors branch:
eu.gcr.io/gardener-project/gardener/dashboard:1.27.0-dev-a99ff03
@holgerkoser related to your query, "What ist the oidc configuration of your api-server?". It's AKS cluster, we don't have control of passing OIDC parameters.
@praveendhac I this case I assume it is not email
but sub
subject. Have you tried to use the value of sub
in the role bindings. Or you have to ask Azure what is the relevant field in the token. The default is sub
!!!
For Roles and Bindings I can use AzureAD Groups/Users as subjects. Should I change this
{{- if not .Values.kubeconfig }}
apiVersion: {{ include "rbacversion" . }}
kind: ClusterRoleBinding
metadata:
name: garden.sapcloud.io:dashboard:admin
labels:
app: gardener-dashboard
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
release: "{{ .Release.Name }}"
heritage: "{{ .Release.Service }}"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: gardener-dashboard
namespace: garden
{{- end }}
to
{{- if not .Values.kubeconfig }}
apiVersion: {{ include "rbacversion" . }}
kind: ClusterRoleBinding
metadata:
name: garden.sapcloud.io:dashboard:admin
labels:
app: gardener-dashboard
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
release: "{{ .Release.Name }}"
heritage: "{{ .Release.Service }}"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: Group
name: azuread-group-user-is-member-of
namespace: garden
{{- end }}
or create a new role/binding with the Group/Users in AzureAD.
I can try different token types which Azure supports.
@praveendhac I think this is the implementation used in AKS. i didnt go through it but seems like @holgerkoser is right about the sub
field.
Also i think this page tells a little bit more about AKS specific implementation https://github.com/Azure/aks-engine/blob/master/docs/kubernetes/aad.md#deployment
@holgerkoser Tried with sub, seeing below error
Other error observed after disabling CSP and CORS on the browser.
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://login.windows.net/common/discovery/keys. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing)
and on the gardener-dashboard logs we can only see below logs, is there a way to enable more verbose debugging that could help us:
2018-12-21T15:45:32.087Z - error: Error with invalid code undefined: Response code 404 (Not Found) JwksError: Response code 404 (Not Found)
at got.then.catch.err (/usr/src/app/lib/middleware.js:82:19)
at process._tickCallback (internal/process/next_tick.js:68:7)
2018-12-21T15:45:32.088Z - http:10.240.0.4 - - [21/Dec/2018:15:45:32 +0000] "GET /api/user HTTP/1.1" 401 117
@bergerx, @praveendhac Sorry for the late answer, we were on vacation over New Year's.
The CORS problem is an AZURE problem. They do not allow CORS for the jwks endpoint https://login.windows.net/common/discovery/keys. As I have already written the following configuration in the helm values.yaml is a workaround for this issue.
oidc:
...
metadata:
...
jwks_uri: /keys
The others error Response code 404 (Not Found) JwksError: Response code 404 (Not Found)
comes from the jwt middleware that validates the token in the backend. It looks like the jwksUri returns 404. You can enable jwks trace logs by setting the environment variable
DEBUG=jwks
in the dashboard backend container.
Regarding the sub
field in the token. How does it look like in the AKS case? You can see it if you take a look at the localstorage in the browser. There is an item which starts with oidc:user:
. The value contains the user profile.sub
. Is it the email of the current user or a token or uid? You can also see the username in the backend logs. There should be lines like this.
2019-01-07T02:30:54.278Z - debug: Socket /shoots#NYyupo9iQhpoDQDcAAFa authenticated (user foo.bar@example.org)
@holgerkoser Added the debug flag in deployments
- name: KUBECONFIG
value: /etc/gardener-dashboard/secrets/kubeconfig/kubeconfig
- name: DEBUG
value: jwks
P.S. I am not seeing the ENV variables in gardener-dashboard Pod and not seeing debug logs either.
values.yaml has below config
oidc:
issuerUrl: https://sts.windows.net/xx-yy
clientId: 2xx-yy9
redirectUri: https://pd-dshboard.ingress.my-domain.io/callback
responseType: 'id_token'
scope: 'sub'
rejectUnauthorized: flase
metadata:
jwks_uri: /keys
Pulled dashboard repo today(10-Jan-2019) and applied above config, clicking the Login tab doesn't take me anywhere and doesn't show up anything in Dev Tools except below information under Console
Strict-Transport-Security: The connection to the site is untrustworthy, so the specified header was ignored.[Learn More] chunk-e8f5e4a6.680c8a47.js
JsonService.getJson: network error oidc-client.min.js:1:2733
dd17/</</e.Log</t.error
oidc-client.min.js:1:2733
dd17/</</e.JsonService</t.prototype.getJson/</s.onerror
oidc-client.min.js:3:4812
signin error Network Error auth.js:45
c/<
auth.js:45
With old dashboard-repo (dated 21-Dec-2018) I get below error with same config and cannot load the UI
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://login.windows.net/common/discovery/keys. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).[Learn More]
JsonService.getJson: network error oidc-client.min.js:1:2733
signinCallback error Network Error auth.js:54
Router error: Network Error index.js:466
None of the parameter in values.yaml points to https://login.windows.net/common/discovery/keys, not sure how we end up using the environment.
@holgerkoser Able to get dashboard UI after disabling CORS, using "CORS Everywhere" Firefox extension. This is the Local Storage for AzureAD login
oidc.user:https://sts.windows.net/my-tenantId:my-clientId:”{“id_token":"ey….A”,”session_state":"4<REDACTED>e","profile":{"aio”:”<REDACTED_base64Data>”,”amr":["pwd"],"email”:”my-loggedin-email”,”hasgroups":"true","idp":"https://sts.windows.net/my-clientId/","in_corp":"true","ipaddr”:”my-Public-IP”,”name”:”loggedin-user-name”,”oid":"3<REDACTED>8","sub":"<REDACTED_base64Data>","tid":"my-tenantId","unique_name":"my-loggedin-email","uti":"<REDACTED_base64Data>","ver":"1.0"}}"
Seeing below exception in gardener-dashboard Pod
2019-01-10T18:24:15.210Z - http:10.244.1.1 - - [10/Jan/2019:18:24:15 +0000] "GET /api/namespaces HTTP/1.1" 401 135
2019-01-10T18:24:15.211Z - error: Error with invalid code credentials_bad_scheme: Format is Authorization: Bearer [token] UnauthorizedError: Format is Authorization: Bearer [token]
at middleware (/usr/src/app/node_modules/express-jwt/lib/index.js:64:25)
at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (/usr/src/app/node_modules/express/lib/router/index.js:317:13)
at /usr/src/app/node_modules/express/lib/router/index.js:284:7
at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:335:12)
at next (/usr/src/app/node_modules/express/lib/router/index.js:275:10)
at cors (/usr/src/app/node_modules/cors/lib/index.js:188:7)
at /usr/src/app/node_modules/cors/lib/index.js:224:17
at originCallback (/usr/src/app/node_modules/cors/lib/index.js:214:15)
at /usr/src/app/node_modules/cors/lib/index.js:219:13
2019-01-10T18:24:15.211Z - http:10.244.1.1 - - [10/Jan/2019:18:24:15 +0000] "GET /api/domains HTTP/1.1" 401 135
2019-01-10T18:24:15.970Z - error: Socket /shoots#32LGCowaIrkybMe5AAAA: no user on response object
2019-01-10T18:24:15.971Z - error: Socket /shoots#32LGCowaIrkybMe5AAAA authentication failed: "Response code 404 (Not Found)"
2019-01-10T18:24:15.997Z - error: Socket /journals#32LGCowaIrkybMe5AAAA: no user on response object
2019-01-10T18:24:15.998Z - error: Socket /journals#32LGCowaIrkybMe5AAAA authentication failed: "Response code 404 (Not Found)"
2019-01-10T18:29:32.171Z - error: watch seeds disconnected { Error
at createError (/usr/src/app/lib/kubernetes/watch.js:38:15)
at WebSocket.onClose (/usr/src/app/lib/kubernetes/watch.js:127:27)
at WebSocket.emit (events.js:182:13)
at WebSocket.emitClose (/usr/src/app/node_modules/ws/lib/websocket.js:172:10)
at TLSSocket.socketOnClose (/usr/src/app/node_modules/ws/lib/websocket.js:781:15)
at TLSSocket.emit (events.js:187:15)
at _handle.close (net.js:610:12)
at TCP.done (_tls_wrap.js:386:7) code: 1006 }
401 errors
@praveendhac The oidc
settings you are using are wrong. It could never work with these settings. Please read the the documentation of the oidc-client
scope
should look like openid email profile groups
. A value of sub
makes no sense. I think it is even not valid. As I have already written in a previous comment. It should look something like this. But please check all details.
oidc:
issuerUrl: &issuer https://sts.windows.net/{tenant}/
clientId: {application_id)
authority: https://login.microsoftonline.com/{tenant}/
redirectUri: https://{your.domain.org}/callback
responseType: 'id_token'
scope: 'openid email profile groups'
loadUserInfo: false
metadata:
issuer: *issuer
authorization_endpoint: https://login.microsoftonline.com/{tenant}/oauth2/authorize
jwks_uri: /keys
sub
field in the RBAC bindings as username because I think this defines the username in the AKS apiServer
--oidc-username-claim="sub"
@holgerkoser I am using AKS, the method you are suggesting is directly passing OIDC params to API Server which is not possible in my case as I don't have any control over API Server.
--oidc-username-claim="sub"
@praveendhac I guess AKS does it differently, see https://github.com/Azure/aks-engine/blob/93caa9ba592d7be5e7d6923ad74adbb2d6348b5c/pkg/api/defaults-apiserver.go#L77
Using below config as suggested by @holgerkoser not seeing any change in UI behaviour.
oidc:
issuerUrl: https://sts.windows.net/{tenant}
# Native
clientId: {application-id}
authority: https://login.microsoftonline.com/{tenant}
redirectUri: https://pd-dshboard.ingress.{redacted}/callback
responseType: 'id_token'
scope: 'openid email profile groups'
rejectUnauthorized: flase
metadata:
issuer: https://sts.windows.net/{tenant}
authorization_endpoint: https://login.microsoftonline.com/{tenant}/oauth2/authorize
jwks_uri: /keys
Debug logs jwks_debug-pd.log
@bergerx I am still unable to figure out --oidc-username-claim="sub"
This is how I am associating authenticated user/email to cluster
$ kubectl get clusterrolebindings pd-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: "2019-01-11T13:12:40Z"
name: pd-admin
resourceVersion: "23186941"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/pd-admin
uid: 92cf4d19-15a2-11e9-bf18-0a58ac1f1a76
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: {my-loggedin-email}
only missing part is sub
.
@holgerkoser we are referring to .Values.kubeconfig
in deployments, what is it's role.
Fri, 11 Jan 2019 13:37:21 GMT jwks Fetching keys from 'https://sts.windows.net/{tenant}/keys'
Fri, 11 Jan 2019 13:37:21 GMT jwks Http Error: { HTTPError: Response code 404 (Not Found)
You get a 404 for the keys url? Have you verified that the url is correct?
@praveendhac The PR #270 with the workaround for CORS problem was not merged. Therefor the with the master branch the oidc.metadata
have been ignored. Now I have merged the PR. The resulting latest dev image is eu.gcr.io/gardener-project/gardener/dashboard:1.27.0-dev-5f9b75d432f11f682629712347fe3a7e54bda9cb
Of course you have to replace the values for {tenant}
and {application-id}
with the one from your azure account. I could not know the concrete values. With the variable names it could not work.
Please do not use .Values.kubeconfig
in you case. This is only neccessary if your dashbaord is deployed in a different kubernetes cluster and cannot use the inCluster configuration.
@bergerx @praveendhac Ok. This means they use the field oid
as username. You should use the value of oid
in the RBAC role binding e.g.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: pd-admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: "3<REDACTED>8"
{tenant}, {application-id}
are replaced with the right values created from Azure portal, can't post them here :-)
Fixed jwks Fetching keys
error using below config
oidc:
jwksUri: https://sts.windows.net/{tenant}/discovery/keys
Added above RBAC with oid
ID, still seeing 401
errors on dashboard URL's, attaching logs for your reference
jwks_debug_pd.log
.
This seems to be another issue,gardener dashboard sends wrong request header Authorization undefined eyXXXXXw
instead of Authorization Bearer eyXXXXXw
2019-01-11T15:38:25.551Z - error: Error with invalid code credentials_bad_scheme: Format is Authorization: Bearer [token] UnauthorizedError: Format is Authorization: Bearer [token]
This seems to be another issue,gardener dashboard sends wrong request header Authorization undefined eyXXXXXw instead of Authorization Bearer eyXXXXXw
where do you see this? in chrome's developer tools?
here is how the authorization header is constructed https://github.com/gardener/dashboard/blob/master/frontend/src/utils/api.js#L27
how can this result in undefined eyXXXXXw
🤔
Yes, it's in developer tools logs, the Authorization
header has undefined
instead of Bearer
. You can also see one of the comments from bergerx.
@praveendhac @bergerx I have merged the PR #268 27 days ago. In this PR I have hardcoded the authorization schema to be Bearer
. @petersutter mentioned this in his last comment (https://github.com/gardener/dashboard/blob/master/frontend/src/utils/api.js#L27). It is not possible that you use the latest dev image eu.gcr.io/gardener-project/gardener/dashboard:1.27.0-dev-5f9b75d432f11f682629712347fe3a7e54bda9cb
I have build for you and still having the authorization schema undefined
.
Authorization: undefined XXXX
issue is gone with eu.gcr.io/gardener-project/gardener/dashboard:1.27.0-dev-5f9b75d432f11f682629712347fe3a7e54bda9cb
. Thank you.
Getting 401
error when dashboard accesses URLs /api/user, /api/cloudprofiles, /api/domains and /api/namespaces
, trace below
2019-01-14T12:31:57.789Z - error: Error with invalid code invalid_token: jwt issuer invalid. expected: https://sts.windows.net/{tenant} UnauthorizedError: jwt issuer invalid. expected: https://sts.windows.net/{tenant}
at /usr/src/app/node_modules/express-jwt/lib/index.js:102:22
at /usr/src/app/node_modules/jsonwebtoken/verify.js:166:16
at getSecret (/usr/src/app/node_modules/jsonwebtoken/verify.js:76:14)
at Object.module.exports [as verify] (/usr/src/app/node_modules/jsonwebtoken/verify.js:80:10)
at verifyToken (/usr/src/app/node_modules/express-jwt/lib/index.js:100:13)
at fn (/usr/src/app/node_modules/async/lib/async.js:746:34)
at /usr/src/app/node_modules/async/lib/async.js:1213:16
at /usr/src/app/node_modules/async/lib/async.js:166:37
at /usr/src/app/node_modules/async/lib/async.js:706:43
at /usr/src/app/node_modules/async/lib/async.js:167:37
2019-01-14T12:31:57.792Z - http:10.240.0.5 - - [14/Jan/2019:12:31:57 +0000] "GET /api/user HTTP/1.1" 401 186
Mon, 14 Jan 2019 12:31:57 GMT jwks Fetching signing key for 'nbCwW11w3XXkB-xUaXyXwKRSuLjMHGQ'
Mon, 14 Jan 2019 12:31:57 GMT jwks Fetching keys from 'https://sts.windows.net/{tenant}/discovery/keys'
Mon, 14 Jan 2019 12:31:57 GMT jwks Fetching signing key for 'nbCwW11w3XXkB-xUaXyXwKRSuLjMHGQ'
Mon, 14 Jan 2019 12:31:57 GMT jwks Fetching keys from 'https://sts.windows.net/{tenant}/discovery/keys'
Mon, 14 Jan 2019 12:31:57 GMT jwks Fetching signing key for 'nbCwW11w3XXkB-xUaXyXwKRSuLjMHGQ'
Mon, 14 Jan 2019 12:31:57 GMT jwks Fetching keys from 'https://sts.windows.net/{tenant}/discovery/keys'
Token looks valid, verified the token on jwt.io Issuer configured in values.yaml
oidc:
issuerUrl: https://sts.windows.net/{tenant}
replace {tenant}
with proper value from Azure Subscription.
The token is valid but your configuration is wrong. The values in the configuration has to exactly match the values in the https://sts.windows.net/{tenantId}/.well-known/openid-configuration. The issuer has trailing slash:
...
"issuer": "https://sts.windows.net/{tenantid}/",
...
If you look carefully in my previous comments you will see that I have proposed the correct URL with trailing slash. You have removed the trailing slash in your config.
oidc:
issuerUrl: https://sts.windows.net/{tenant}/
clientId: {application_id)
authority: https://login.windows.net/{tenant}/
redirectUri: https://{your.domain.org}/callback
responseType: 'id_token'
scope: 'openid email profile groups'
loadUserInfo: false
metadata:
issuer: https://sts.windows.net/{tenant}/
authorization_endpoint: https://login.windows.net/{tenant}/oauth2/authorize
jwks_uri: /keys
@holgerkoser Thanks for the input, all the errors are gone but the link to CREATE PROJECT
is grayed out. Has proper RBAC's configured for the logged in users. RBAC's are based on email and oid with admin
role.
Works if email/oid is associated with cluster-admin
ClusterRole.
Still throws CORS errors with default browser settings with image eu.gcr.io/gardener-project/gardener/dashboard:1.27.0-dev-5f9b75d432f11f682629712347fe3a7e54bda9cb
@praveendhac This is not possible. If you use the latest image and your configuration contains
metadata:
issuer: https://sts.windows.net/{tenant}/
authorization_endpoint: https://login.windows.net/{tenant}/oauth2/authorize
jwks_uri: /keys
What is the resulting frontend configuration of the dashboard. What is the json body of this URL https://pd-dshboard.ingress.{your.domain}/config.json
. The oidc property is important. Does it contain a metadata property? What is the value of oidc.metadata.jwks_uri
?
As I have already written before. The CORS problem has nothing to do with gardener dashboard but it is a bug of the Azure Active Directory OIDC implementation from my point of view. I have provided a workaround which proxies the jwks endpoint from the gardener-dashboard backend to circumvent the Azure CORS problem. As I have written previously the oidc-client we are using allows to specify the oidc metadata directly https://github.com/IdentityModel/oidc-client-js/wiki#provider-settings-if-cors-not-supported-on-oidcoauth2-provider-metadata-endpoint. If the frontend config contain oidc.metadata.jwks_uri
it will use this value and not the value from https://sts.windows.net/{tenantId}/.well-known/openid-configuration. I hope you have NOT used the the value https://login.windows.net/common/discovery/keys
for jwks_uri
, which would explain the behavior.
Below config (helms values.yaml) worked for us.
$ cat ../../gardener-dashboard-working-values.yaml
# Default values for gardener-dashboard.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
image:
repository: eu.gcr.io/gardener-project/gardener/dashboard
tag: 1.27.0-dev-5f9b75d432f11f682629712347fe3a7e54bda9cb
pullPolicy: Always
logLevel: trace
apiServerUrl: https://{k8s-api-server}
containerPort: 8080
servicePort: 8080
resources:
limits:
cpu: 250m
memory: 256Mi
requests:
cpu: 100m
memory: 128Mi
hosts:
- pd-dshboard.ingress.example.com
tls:
crt: |
-----BEGIN CERTIFICATE-----
Li4u
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
Li4u
-----END CERTIFICATE-----
key: |
-----BEGIN RSA PRIVATE KEY-----
Li4u
-----END RSA PRIVATE KEY-----
jwks:
jwksRequestsPerMinute: 5
jwksUri: /keys
oidc:
issuerUrl: https://sts.windows.net/{tenand-id}/
# Webapp registered in AzureAD App-registration
clientId: {application-id}
authority: https://login.microsoftonline.com/{tenand-id}/
redirectUri: https://pd-dshboard.ingress.example.com/callback
responseType: 'id_token'
scope: 'openid email profile groups'
jwksUri: https://sts.windows.net/{tenand-id}/discovery/keys
rejectUnauthorized: flase
metadata:
issuer: https://sts.windows.net/{tenand-id}/
authorization_endpoint: https://login.microsoftonline.com/{tenand-id}/oauth2/authorize
jwks_uri: /keys
frontendConfig:
landingPageUrl: https://github.com/gardener
helpMenuItems:
- title: Getting Started
icon: description
url: https://github.com/gardener/gardener/tree/master/docs
- title: Issues
icon: bug_report
url: https://github.com/gardener/gardener/issues
gitHubRepoUrl: https://foo-github.com/dummyorg/dummyrepo
prometheus:
secret: secret
livenessProbe:
enabled: true
initialDelaySeconds: 15
periodSeconds: 20
timeoutSeconds: 5
failureThreshold: 6
successThreshold: 1
readinessProbe:
enabled: true
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 6
successThreshold: 1
replace pd-dshboard.ingress.example.com
with your dashboards FQDN/host.
To see jwks debug logs in gardener-dashboard
pod add following config to charts/gardener-dashboard/templates/deployment.yaml
env:
- name: DEBUG
value: jwks
You need to create clusterrolebinding
to give required permissions to users logging in to dashboard to read projects, get namespaces, read secrets etc.
$ cat pd-cadmin-dashboard.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: pd-cadmin-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: {user-email-inAzureAD}
- apiGroup: rbac.authorization.k8s.io
kind: User
name: {user-email-ObjecdID-inAzureAD}
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: {AzureAD-group-name}
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: {AzureAD-GroupID-aka-ObjectID}
There are CORS errors and gardener-dashboard
UI doesn't load. I used CORS Everywhere
Firefox browser plugin to suppress CORS, and couple of UI refreshes to load the UI.
TL;DR: OAuth parameter for AKS kube-apiserver's OIDC parameters are not known, and we are not able to configure the Gardener Dashboard the correct way.
We are hosting our Garden and Seed clusters on AKS as described in docs/deployment/aks.md. But with one change. In our Garden AKS cluster now we have AAD enabled. After we wrote that doc AKS went GA and they enabled RBAC and AAD integrations. I created https://github.com/gardener/gardener/issues/551 to update the AKS deployment doc.
Previously we were not able to do that since the we had no control over AKS OIDC parameters. We were not able to deploy the Gardener Dashboard since it requires the Garden Kubernetes cluster authentication work with Oauth token. After AKS went GA and they enabled AAD and RBAC on AKS clusters (https://docs.microsoft.com/en-us/azure/aks/aad-integration), this now enabled us deploy the Dashboard on top AKS. At least we thought that it would.
We enabled AAD on our AKS Garden clusters and started investigating how to configure the Gardener Dashboard to work with AKS. But the Dashboard needs to be configured with the same OIDC parameters with the underlying Kubernetes cluster. But we were on AKS and we don't have visibility on control plane components configuration including the kube-apiserver of the cluster which has the OIDC configuration.
We tried to configure the Gardener with whatever we are able to find in the https://docs.microsoft.com/en-us/azure/aks/aad-integration page create a compatible OIDC parameters for Gardener Dashboard. But we were not able to be successful figure out the right configuration values for the Dashboard.
Next, we tried putting DEX as an intermediary server, Dashboard was using Dex and DEX was authenticating us to the same AAD with the AKS. This time we were able to login to the Dashboard but we were not able to see any resources. This is because the Auth Token provided by DEX is actually not valid in AKS, DEX was kind of proxying the Token.
After spending some time dealing with AAD and DEX configuration we gave up.