Metrics endpoint does not configure correctly in logging CRD

Problem description

Using the "logging" CRD, I'm noticing that the fluentd and fluentbit configurations are not respecting the metrics.serviceMonitor: true configuration. Interestingly, it does configure the buffer metrics for fluentd.

What I've noticed is that with metrics enabled, the operator will configure the serviceMonitor objects in K8s but do not configure the metrics endpoints on the daemonset for fluentbit nor the statefulset for fluentd.

Versions: Logging-operator helm chart version: 4.9.0 Logging-operator docker image: ghcr.io/kube-logging/logging-operator:4.9.0

"logging" manifest":

---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: logging
spec:
  controlNamespace: logging-operator
  fluentbit:
    inputTail:
      Mem_Buf_Limit: 50M
    metrics:
      serviceMonitor: true
    podPriorityClassName: system-node-critical
    resources:
      limits:
        cpu: 500m
        memory: 200Mi
      requests:
        cpu: 250m
        memory: 100Mi
    tolerations: []
  fluentd:
    bufferStorageVolume:
      pvc:
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi
          storageClassName: ebs-csi-gp3-persistent
          volumeMode: Filesystem
    bufferVolumeMetrics:
      serviceMonitor: true
    logLevel: info
    metrics:
      serviceMonitor: true
    podPriorityClassName: system-node-critical
    resources:
      limits:
        cpu: 2
        memory: 300Mi
      requests:
        cpu: 500m
        memory: 100Mi
    scaling:
      replicas: 3
    tolerations: []

Screenshots: The serviceMonitor is created as expected

However, the http-metrics port is not exposed in the statefulset configuration

Any help or advice is much appreciated

Hey @ServerNinja,

Thanks for using Logging-operator. Here's how our monitoring system works:

When metrics.serviceMonitor is enabled, the operator creates a Kubernetes Service with the suffix "-metrics" that exposes the configured metrics endpoint of your SyslogNG/Fluentd/Fluentbit pods.

At the same time a Service monitor is created which points to the services port with the name described in the Service monitor manifest (Match labels are also being used to pin-point the right service):

On the pods side, we have the exposed endpoint but with the name "monitor".

The Prometheus Service monitor will be able to scrape the metrics this way because the port numbers and the endpoint names match.
Manually tested:

kube-logging / logging-operator

Metrics endpoint does not configure correctly in logging CRD #1830