500 Internal Error Additional Trusted CA Bundle missing

[gfvirga]

Hello,

The monitoring operator is not adding the additionalTrustBundle from installation into the proxy sidecar containers:

• alertmanager-proxy
• prometheus-proxy
• grafana-proxy

This is similar to the issue I just opened https://github.com/openshift/cluster-logging-operator/issues/261

Example: The prometheus sidecar pod does not have the configmap for ca-trust bundle.

$ oc rsh -c prometheus-proxy prometheus-k8s-1
sh-4.2$ curl https://oauth-openshift.apps.ose.company.com/ 
curl: (60) Peer's Certificate issuer is not recognized.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
sh-4.2$ 

It seems that alertmanager-trusted-ca-bundle configmap is created, but not configured in the sidecar proxy pod.

:~/openshift-install/configs/efk$ oc get configmaps | grep ca
alertmanager-trusted-ca-bundle                        1      8d
alertmanager-trusted-ca-bundle-f7rp335d3d8u8          1      8d
kubelet-serving-ca-bundle                             1      8d
serving-certs-ca-bundle                               1      8d
telemeter-client-serving-certs-ca-bundle              1      8d
telemeter-trusted-ca-bundle                           1      8d
telemeter-trusted-ca-bundle-f7rp335d3d8u8             1      8d
trusted-ca-bundle                                     1      8m2s  ( CREATED MANUALLY BY ME)

I am manually changing the operators to unmanaged and then editing the deployments to use the configmaps.

Is there a better way to do this?

Thank you, Gabe Virga

[gfvirga]

Found the https://github.com/openshift/cluster-monitoring-operator/pull/448 that is adding the trust bundle for alertmanager, but it was not implemented in https://github.com/openshift/cluster-monitoring-operator/blob/release-4.2/assets/alertmanager/alertmanager.yaml Should the configuration be added elsewhere?

[lilic]

Thanks for your issue! :) I answered the question on the PR about alertmanager pods itself. As for the proxy ones cc @paulfantom should have more info that.

[gfvirga]

https://bugzilla.redhat.com/show_bug.cgi?id=1766181

[rinormaloku]

I am manually changing the operators to unmanaged and then editing the deployments to use the configmaps.

Hi Gabe,

I am facing the same issue but I cannot find how to switch the operator to unmanaged in OpenShift 4.2, is it possible?

[gfvirga]

Under Custom Resource Definitions. Have to replace the registry with monitoring stuff. $ oc edit cluster version ... spec: overrides:

• group: apps/v1 kind: Deployment name: cluster-image-registry-operator namespace: openshift-image-registry unmanaged: true

I was only able to fix grafana, the others staid controlled by the operator :(. I am on vacation now away from my computer :(

[gfvirga]

Why is this closed, is it fixed?

[s-urbaniak]

@gfvirga yes, this has been fixed in 4.3.0 as per https://bugzilla.redhat.com/show_bug.cgi?id=1766181