Hi,
I have deployed the application-monitoring-operator to a OpenShift 4.3 cluster built on-prem. All components seem to work fine, without errors. However, I get error "500 Internal Error" right after I try to login to Grafana UI or to Prometheus UI. We use internally signed certificates and custom Certification Authority in our OpenShift environment.
I click "Login with OpenShift" and get redirected to the OpenShift login, then I insert my username and password, I authorize the application to read my information, I get "500 Internal Error" message on the page.
Try steps 2 and 3 for Prometheus UI and also get "500 Internal Error".
Environment info:
oc get pods
NAME READY STATUS RESTARTS AGE
alertmanager-application-monitoring-0 3/3 Running 0 59m
application-monitoring-operator-5bc879f697-mcglx 1/1 Running 0 60m
grafana-deployment-58746b4f54-hr4xs 2/2 Running 0 9m32s
grafana-operator-66497b6fc6-q9lhc 1/1 Running 0 59m
prometheus-application-monitoring-0 5/5 Running 1 59m
prometheus-operator-76b4dfbb68-r7k95 1/1 Running 0 59m
I also tried to edit Grafana and Prometheus instances of CRDs. However, in all cases the configuration is ovewritten by operator, which is expected behaviour, I believe.
Please advice, what is the correct flow of adding trusted-ca-bundle with this operator?
Thank you!
Sergiy
Hi, I have deployed the application-monitoring-operator to a OpenShift 4.3 cluster built on-prem. All components seem to work fine, without errors. However, I get error "500 Internal Error" right after I try to login to Grafana UI or to Prometheus UI. We use internally signed certificates and custom Certification Authority in our OpenShift environment.
Steps to reproduce:
and wait for completion. Components install without errors.
Environment info: oc get pods NAME READY STATUS RESTARTS AGE alertmanager-application-monitoring-0 3/3 Running 0 59m application-monitoring-operator-5bc879f697-mcglx 1/1 Running 0 60m grafana-deployment-58746b4f54-hr4xs 2/2 Running 0 9m32s grafana-operator-66497b6fc6-q9lhc 1/1 Running 0 59m prometheus-application-monitoring-0 5/5 Running 1 59m prometheus-operator-76b4dfbb68-r7k95 1/1 Running 0 59m
Logs for grafana-proxy container:
2020/06/01 10:18:45 provider.go:117: Defaulting client-id to system:serviceaccount:application-monitoring:grafana-serviceaccount 2020/06/01 10:18:45 provider.go:122: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token 2020/06/01 10:18:45 provider.go:310: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates. 2020/06/01 10:18:45 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:3000/" 2020/06/01 10:18:45 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics" 2020/06/01 10:18:45 oauthproxy.go:227: OAuthProxy configured for Client ID: system:serviceaccount:application-monitoring:grafana-serviceaccount 2020/06/01 10:18:45 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain: refresh:disabled
2020/06/01 10:18:45 http.go:106: HTTPS: listening on [::]:9091
2020/06/01 10:21:25 provider.go:392: authorizer reason:
2020/06/01 10:21:28 provider.go:573: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2020/06/01 10:21:28 provider.go:613: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
"issuer": "https://oauth-openshift.apps.os4-test.lab.local",
"authorization_endpoint": "https://oauth-openshift.apps.os4-test.lab.local/oauth/authorize",
"token_endpoint": "https://oauth-openshift.apps.os4-test.lab.local/oauth/token",
"scopes_supported": [
"user:check-access",
"user:full",
"user:info",
"user:list-projects",
"user:list-scoped-projects"
],
"response_types_supported": [
"code",
"token"
],
"grant_types_supported": [
"authorization_code",
"implicit"
],
"code_challenge_methods_supported": [
"plain",
"S256"
]
}
2020/06/01 10:21:38 provider.go:573: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2020/06/01 10:21:38 provider.go:613: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
"issuer": "https://oauth-openshift.apps.os4-test.lab.local",
"authorization_endpoint": "https://oauth-openshift.apps.os4-test.lab.local/oauth/authorize",
"token_endpoint": "https://oauth-openshift.apps.os4-test.lab.local/oauth/token",
"scopes_supported": [
"user:check-access",
"user:full",
"user:info",
"user:list-projects",
"user:list-scoped-projects"
],
"response_types_supported": [
"code",
"token"
],
"grant_types_supported": [
"authorization_code",
"implicit"
],
"code_challenge_methods_supported": [
"plain",
"S256"
]
}
2020/06/01 10:21:38 oauthproxy.go:645: error redeeming code (client:10.254.3.1:47476): Post https://oauth-openshift.apps.os4-test.lab.local/oauth/token: x509: certificate signed by unknown authority
2020/06/01 10:21:38 oauthproxy.go:438: ErrorPage 500 Internal Error Internal Error
So I understand that the issue is with internally signed certificates.
I added two configmaps with our root certificates and labels: config.openshift.io/inject-trusted-cabundle: 'true'
I tried to add two sections to grafana-deployment: volumeMounts:
I also tried to edit Grafana and Prometheus instances of CRDs. However, in all cases the configuration is ovewritten by operator, which is expected behaviour, I believe.
Please advice, what is the correct flow of adding trusted-ca-bundle with this operator? Thank you! Sergiy