search

3scale-ops / prometheus-exporter-operator

Operator to centralize the setup of 3rd party prometheus exporters on Kubernetes/OpenShift, with a collection of grafana dashboards
Apache License 2.0
42 stars 15 forks source link

Disable kube-rbac-proxy from prometheus-exporter-operator controller-manager #26

Closed slopezz closed 3 years ago

slopezz commented 3 years ago

kube-rbac-proxy is used by default on operator-sdk to protect operator metrics path, in case you don't want anyone in the cluster but only k8s authenticated resources can access to them.

But there are 2 current issues with kube-rbac-proxy:

  1. Upgrade from kube-rbac-proxy:v0.5.0 to latest kube-rbac-proxy:v0.8.0 https://github.com/operator-framework/operator-sdk/issues/4684 causes error on OpenShift 4.6+: 
    Error: container create failed: time="2021-03-19T15:51:12Z" level=error msg="container_linux.go:366: starting container     process caused: chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"
    • The current solution is to use a different proxy image on OpenShift 4.6+ openshift4/ose-kube-rbac-proxy:v4.7.0 (which works OK), but this image is behind registry.redhat.io registry which requires authenticated if not using Openshift (it doesn't work directly on vanilla k8s), so you need to maintain 2 different bundles with different proxy images if you want to run the operator on both OpenShift or K8s, which makes maintenance more complex.
  2. In addition, OpenShift User Workload Monitoring (OpenShift official monitoring stack) does not support ServiceMonitor with bearerTokenFile field https://github.com/operator-framework/operator-sdk/issues/4764#issuecomment-821264658 (which is needed to scrape metrics behind kube-rbac-proxy) , so it seems there is no way of having operator metrics with auth if using OCP UWM.

For that reason, taking into account that operator metrics are not that important to have them with forced auth, I have disabled kube-rbac-proxy container (making a few changes to make that work with a new patch, and leaving default proxy yamls there, in case they want to be enabled easily in the future), so anyone once inside the cluster could check operator metrics without any problem on both OCP and k8s (even if using OCP UWM).

slopezz commented 3 years ago

/kind feature /priority important-soon /label size/xs /assign

roivaz commented 3 years ago

/lgtm

3scale-robot commented 3 years ago

LGTM label has been added.

Git tree hash: 81832963d343714fa6597e71a85f79fc36b6afa0

raelga commented 3 years ago

/lgtm

raelga commented 3 years ago

/shrug

slopezz commented 3 years ago

/approve

3scale-robot commented 3 years ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: slopezz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/3scale-ops/prometheus-exporter-operator/blob/main/OWNERS)~~ [slopezz] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment