search

rhobs / observability-operator

Create Monitoring Stacks you need on your Kubernetes clusters!
32 stars 56 forks source link

MonitoringStack errors on quota for thanos-sidecar pod #528

Open grigassukys opened 3 months ago

grigassukys commented 3 months ago

Hello,

We have an issue with Cluster Observability Operator, in particular when creating MonitoringStack. For example when using this yaml:

apiVersion: monitoring.rhobs/v1alpha1
kind: MonitoringStack
metadata:
  labels:
    mso: test-prometheus
  name: test-monitoring-stack
  namespace: test-monitoring
spec:
  alertmanagerConfig:
    disabled: true
  namespaceSelector:
    matchExpressions:
      - values:
          - test-monitoring
        key: kubernetes.io/metadata.name
        operator: In
  logLevel: debug
  resourceSelector:
    matchLabels:
      app: test-metrics
  retention: 1d

The Monitoring Stack is created and in Resources tab we can see:

"test-monitoring-stack-prometheus"
"test-monitoring-stack-prometheus-additional-scrape-configs"
"test-monitoring-stack-thanos-sidecar"

The problem is, that prometheus pods cannot start, because we get the following error: "create Pod prometheus-test-monitoring-stack-0 in StatefulSet prometheus-test-monitoring-stack failed error: pods "prometheus-test-monitoring-stack-0" is forbidden: failed quota: test-monitoring-quota-compute-resources: must specify limits.cpu for: thanos-sidecar; limits.memory for: thanos-sidecar; requests.cpu for: thanos-sidecar; requests.memory for: thanos-sidecar"

We cannot find any documentation/instructions/guide on how to setup the limits for thanos-sidecar, because changing "resources" in the monitoringstack yaml only changes the limits for PROMETHEUS pods, but not thanos-sidecar.

Additionally, in our case we don't even need thanos-sidecar and would like to disable it completely like we can disable "alertmanager".

Thank you!

simonpasquier commented 1 week ago

You should be able to patch the Prometheus resource controlled by the MonitoringStack resource following this page: https://github.com/rhobs/observability-operator/blob/main/docs/user-guides/server-side-apply.md

Additionally, in our case we don't even need thanos-sidecar and would like to disable it completely like we can disable "alertmanager".

The reason why COO automatically deploys a Thanos sidecar is to support integration with the ThanosQuerier.