ServiceMonitor: Invalid Configuration

[rbaumgar]

On my OpenShift 4.16 with OpenShift Lightspeed Operator 0.1.2

ServiceMonitor lightspeed-app-server-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits ServiceMonitor lightspeed-operator-controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via tls config which Prometheus specification prohibits

$ oc get event -n openshift-lightspeed 
LAST SEEN   TYPE      REASON                 OBJECT                                                                  MESSAGE
29m         Warning   InvalidConfiguration   servicemonitor/lightspeed-app-server-monitor                            ServiceMonitor lightspeed-app-server-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits
29m         Warning   InvalidConfiguration   servicemonitor/lightspeed-operator-controller-manager-metrics-monitor   ServiceMonitor lightspeed-operator-controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via tls config which Prometheus specification prohibits

$ oc get servicemonitors.monitoring.coreos.com -n openshift-lightspeed -o yaml|oc neat
- apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    labels:
      app.kubernetes.io/component: metrics
      app.kubernetes.io/managed-by: lightspeed-operator
      app.kubernetes.io/name: lightspeed-service-api
      app.kubernetes.io/part-of: openshift-lightspeed
      monitoring.openshift.io/collection-profile: full
    name: lightspeed-app-server-monitor
    namespace: openshift-lightspeed
  spec:
    endpoints:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      interval: 30s
      path: /metrics
      port: https
      scheme: https
      tlsConfig:
        caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
        certFile: /etc/prometheus/secrets/metrics-client-certs/tls.crt
        keyFile: /etc/prometheus/secrets/metrics-client-certs/tls.key
        serverName: lightspeed-app-server.openshift-lightspeed.svc
    jobLabel: app.kubernetes.io/name
    selector:
      matchLabels:
        app.kubernetes.io/component: application-server
        app.kubernetes.io/managed-by: lightspeed-operator
        app.kubernetes.io/name: lightspeed-service-api
        app.kubernetes.io/part-of: openshift-lightspeed
- apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    labels:
      app.kubernetes.io/component: metrics
      app.kubernetes.io/created-by: lightspeed-operator
      app.kubernetes.io/instance: controller-manager-metrics-monitor
      app.kubernetes.io/managed-by: kustomize
      app.kubernetes.io/name: servicemonitor
      app.kubernetes.io/part-of: lightspeed-operator
      control-plane: controller-manager
      olm.managed: "true"
    name: lightspeed-operator-controller-manager-metrics-monitor
    namespace: openshift-lightspeed
  spec:
    endpoints:
    - path: /metrics
      port: metrics
      scheme: https
      tlsConfig:
        caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
        certFile: /etc/prometheus/secrets/metrics-client-certs/tls.crt
        insecureSkipVerify: false
        keyFile: /etc/prometheus/secrets/metrics-client-certs/tls.key
        serverName: lightspeed-operator-controller-manager-service.openshift-lightspeed.svc
    selector:
      matchLabels:
        control-plane: controller-manager

[raptorsun]

Thank you for raising the issue. Could you please share the OLSConfig CR that produces this problem? I cannot reproduce the problem with Openshift 4.16.

Meanwhile, please try upgrade to version 0.1.3 to see whether this issue persists.

[xiormeesh]

I found this thread because I was fixing the same issue in another project on 4.16, it's caused by .spec.endpoints[] bearerTokenFile being deprecated in 4.16, however it should give a warning for now and not block installation, maybe there is something forcing fails on deprecations on OP's cluster.

[rbaumgar]

In the meantime, the cluster upgraded to version 0.1.3. Alert didn't go away. And yes, it is not blocking, it is found by an alert, PrometheusOperatorRejectedResources