Closed rvansa closed 6 years ago
So I've figured out that
1) the login/password directly provided in oauth_proxy won't get me there, I should click the 'Log in with OpenShift' button (which was spitting error message about missing params).
2) While I was tunneling the connection to lab through ssh (-L9091:prometheus-k8s.openshift-monitoring.svc.cluster.local:9091
) it gives me not-approved callback upon oauth/start
. I need to add <ip> prometheus-k8s-openshift-monitoring.router.default.svc.cluster.local
to my /etc/hosts
and access that one.
Now, trying to get in 'as service' works when I do curl ... -H "Authorization: Bearer $(oc whoami -t)"
, so as a cluster admin. However obtaining some service token, e.g. oc sa get-token grafana
gives me much longer token than oc whoami -t
and I am not let in. Grafana UI works (that's why I've tried to use its token).
And one more question: where do I set the list of approved callbacks? I don't see anything explicit in oauth_proxy arguments as seen in oc describe po prometheus-k8s-0
.
@rvansa can you post your oauth-proxy pod configuration and some more details on the exact steps you are taking to log in? -redirect-url is how you can change the callback, however if you are correctly using a proxy SA that has an associated oauth client you should not normally need to set it.
@mrogers950 This is the proxy container configuration, the prometheus is running in the same pod, having 9090 open:
- args:
- -provider=openshift
- -https-address=:9091
- -http-address=
- -email-domain=*
- -upstream=http://localhost:9090
- -htpasswd-file=/etc/proxy/htpasswd/auth
- -openshift-service-account=prometheus-k8s
- '-openshift-sar={"resource": "namespaces", "verb": "get"}'
- '-openshift-delegate-urls={"/": {"resource": "namespaces", "verb": "get"}}'
- -tls-cert=/etc/tls/private/tls.crt
- -tls-key=/etc/tls/private/tls.key
- -client-secret-file=/var/run/secrets/kubernetes.io/serviceaccount/token
- -cookie-secret-file=/etc/proxy/secrets/session_secret
- -openshift-ca=/etc/pki/tls/cert.pem
- -openshift-ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
- -skip-auth-regex=^/metrics
image: registry.redhat.io/openshift3/oauth-proxy:v3.11
imagePullPolicy: IfNotPresent
name: prometheus-proxy
ports:
- containerPort: 9091
name: web
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/tls/private
name: secret-prometheus-k8s-tls
- mountPath: /etc/proxy/secrets
name: secret-prometheus-k8s-proxy
- mountPath: /etc/proxy/htpasswd
name: secret-prometheus-k8s-htpasswd
I was tunneling the connection through ssh myserver -L9091:prometheus-k8s.openshift-monitoring.svc.cluster.local:9091
(tunneling to service address), and when I access https://localhost:9091/
in the browser I can see the login page with 'Log in with OpenShift' and credentials form. Sending the credentials form gives me the same screen again, though, without any errors. When I press the 'Log in with Openshift' I am redirected to
https://myserver.foo.bar.com:8443/oauth/authorize?approval_prompt=force&client_id=system%3Aserviceaccount%3Aopenshift-monitoring%3Aprometheus-k8s&redirect_uri=https%3A%2F%2Flocalhost%3A9091%2Foauth%2Fcallback&response_type=code&scope=user%3Ainfo+user%3Acheck-access&state=95ec5aa924e0d6c1a773d63d633bfe32%3A%2F
(see that the redirect uri is localhost:9091) which gives me this:
{"error":"invalid_request","error_description":"The request is missing a required parameter, includes an invalid parameter value, includes a parameter more than once, or is otherwise malformed.","state":"4435a90ee62810bff60c24c9c4a567df:/"}
After adding the server's IP to my /etc/hosts and accessing https://prometheus-k8s-openshift-monitoring.router.default.svc.cluster.local
(this is the router address), I can open the login page with 'Log in with OpenShift' and credentials form (see https://imgur.com/a/SyCvPmz ). Sending the credentials form does not work either and gives me the same screen again. Log in with Openshift works on this screen.
From within the cluster (no ssh tunneling), I've tried to do - following the redirects:
> curl -k -v https://prometheus-k8s.openshift-monitoring.svc.cluster.local:9091/oauth/start?rd=/
> curl -k -v "https://myserver.foo.bar.com:8443/oauth/authorize?approval_prompt=force&client_id=system%3Aserviceaccount%3Aopenshift-monitoring%3Aprometheus-k8s&redirect_uri=https%3A%2F%2Fprometheus-k8s.openshift-monitoring.svc.cluster.local%3A9091%2Foauth%2Fcallback&response_type=code&scope=user%3Ainfo+user%3Acheck-access&state=df83fd4eac4cc71a878daeedce33449f%3A%2F"
{"error":"invalid_request","error_description":"The request is missing a required parameter, includes an invalid parameter value, includes a parameter more than once, or is otherwise malformed.","state":"df83fd4eac4cc71a878daeedce33449f:/"}
The second problem I have is when I try to connect as a service account: Since pre-configured grafana is getting the data from prometheus, I've tried to use its SA for the test:
> oc sa get-token grafana
eyJhbGci...(long hash redacted)...7MErz7KtiA
> curl -k -v https://prometheus-k8s-openshift-monitoring.router.default.svc.cluster.local -H 'Authorization: Bearer eyJhbGci...(long hash redacted)...7MErz7KtiA
<gives me the oauth_proxy login page>
> oc whoami -t
<pretty short hash>
> curl -k -v https://prometheus-k8s-openshift-monitoring.router.default.svc.cluster.local -H 'Authorization: Bearer <pretty short hash>'
works!
So now I don't understand what keys could Grafana be using:
> oc describe sa grafana
... (redacted)
Tokens: grafana-token-fjrvf
grafana-token-sph8s
> oc get secret grafana-token-fjrvf -o custom-columns=:data.token
ZXlKaGJHY2lPaUpTV...(long hash redacted)...jFxdGw3TUVyejdLdGlB
> oc get secret grafana-token-sph8s -o custom-columns=:data.token
ZXlKaGJHY2lPaUpTV...(long hash redacted)...iUE8zM0xjck5FajV3Sk5FN2d3
Note that the above hashes are different from what oc sa get-token grafana
gave me but neither of them works when I pass them as the Authorization: Bearer <token>
.
I have nearly the same problem. Can not get it to work with oauth-proxy. I have the same error message
https://s-openshift.mycompany.com/oauth/authorize?approval_prompt=force&client_id=system%3Aserviceaccount%3Aopenshift-monitoring%3Aprometheus-k8s&redirect_uri=https%3A%2F%2Fprometheus-xxx.s-apps.cloud.mycompany.com%2Foauth%2Fcallback&response_type=code&scope=user%3Ainfo+user%3Acheck-access&state=4288120771568032b9a4a1fe69160695%3A%2F
{"error":"invalid_request","error_description":"The request is missing a required parameter, includes an invalid parameter value, includes a parameter more than once, or is otherwise malformed.","state":"df83fd4eac4cc71a878daeedce33449f:/"}
My assumption that Grafana has the needed RBAC was incorrect. In fact it has no rights and when I've accessed it (as admin) it impersonated me accessing Prometheus. That's why the tokens did not work.
The tokens that I get using oc get secret ...
like ZXlKaGJHY2lPaUpTV...
are base64 encoded, if I want to use those I would have to decode them.
@Reamer There's a good chance that this is caused by the redirect_uri
as well. Try to access it using the url that you get from oc get route prometheus-k8s -n openshift-monitoring
. Not sure how to change it if you want to use different domain for access.
My assumption that Grafana has the needed RBAC was incorrect. In fact it has no rights and when I've accessed it (as admin) it impersonated me accessing Prometheus. That's why the tokens did not work.
@rvansa Did you get it to work? Did you give Grafana any RBAC? If yes, which and how? I have a similar problem described here: https://github.com/openshift/oauth-proxy/issues/106
Using reverse proxy for OpenShift AuthN is very useful but I think integration of standard tools like Grafana or Prometheus federation (Prometheus server target) are missing or not supported yet.
I managed to connect from Grafana to default OpenShift Prometheus deployed by the Operator:
htpasswd -s -c <file> <username>
echo -n "new user created" | base64
oc -n openshift-monitoring edit secret prometheus-k8s-htpasswd
Hi,
I'm encoutering this issue after redeploying all certificates inside my cluster. Then the certs of Grafana/Prometheus were not working anymore. Have solved it with below doc https://github.com/openshift/openshift-ansible/pull/11472/commits/e24cee252ce6551c6dbf277009fa826abe7b47ce
Now I can access Grafana dashboard and it pulls data from Prometheus, however Prometheus dashboard is still not accessible
{"error":"invalid_request","error_description":"The request is missing a required parameter, includes an invalid parameter value, includes a parameter more than once, or is otherwise malformed.","
@aliouba What about the value that's in the secret? Shouldn't we use the htpasswd file that's being used from the oc cluster? Have been trying your technique and then my Grafana can't pull data anymore from prometheus
@aliouba I stumbled upon this issue now, where do i store the htpasswd file ? , i already have a htpasswd file for openshift and configured my authentication with it in master-config.yml. I still get the error when visiting the grafana route, should i base64 encode the whole user content in the htpasswd file ?
I have a fresh install of Openshift Enterprise 3.11 which secures access to a Prometheus instance with oauth2_proxy, in prometheus-k8s-X/prometheus-proxy. I am trying to login in with credentials for admin (with cluster-admin role) but upon sending those I just get HTTP response 200 and the same login screen. The log shows me that I wasn't allowed:
But when I have enabled audit log on the master API it seems that I should be:
Where is the problem?