dora-metrics / pelorus

Automate the measurement of organizational behavior
https://pelorus.readthedocs.io/
Apache License 2.0
245 stars 83 forks source link

Promethus operator fails to start Promethus pods after upgrading from pelorus-operator.v0.0.8 to pelorus-operator.v0.0.9 #1140

Closed achdmbp closed 4 months ago

achdmbp commented 4 months ago

OpenShift version

4.13

Problem description

After upgrading from pelorus-operator.v0.0.8 to pelorus-operator.v0.0.9, Promethus operator pod shows below error and fails to start Promethus pods:

level=info ts=2024-07-01T15:26:40.289390758Z caller=operator.go:1163 component=prometheusoperator msg="sync prometheus" key=pelorus-operator/prometheus-pelorus
level=error ts=2024-07-01T15:26:40.322958155Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="Sync \"pelorus-operator/prometheus-pelorus\" failed: creating config failed: generating config failed: unmarshalling additional scrape configs failed: yaml: line 7: did not find expected '-' indicator"

Steps to reproduce

kind: Pelorus
apiVersion: charts.pelorus.dora-metrics.io/v1alpha1
metadata:
  name: pelorus-devspaces
spec:
  federate_openshift_monitoring:
    enabled: true
    metrics_filter:
      # pull devspaces metrics we're interested in
      - "{namespace=~'.*-devspaces',pod=~'workspace.+'}"
  exporters:
    instances: []
  prometheus_storage: true
  prometheus_storage_pvc_capacity: 3Gi
  prometheus_storage_pvc_storageclass: gp2
  prometheus_retention_size: 2GB

Current behavior

Promethus operator pod shows below error and fails to start Promethus pods:

level=info ts=2024-07-01T15:26:40.289390758Z caller=operator.go:1163 component=prometheusoperator msg="sync prometheus" key=pelorus-operator/prometheus-pelorus
level=error ts=2024-07-01T15:26:40.322958155Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="Sync \"pelorus-operator/prometheus-pelorus\" failed: creating config failed: generating config failed: unmarshalling additional scrape configs failed: yaml: line 7: did not find expected '-' indicator"

Expected behavior

Promethus server starts up without errors

Code of Conduct

etsauer commented 4 months ago

@achdmbp I've been able to reproduce your issue. gonna dig into it more now.

etsauer commented 4 months ago

@achdmbp could you try changing your config to (all I did was change the single and double quotes on the metrics_filter):

kind: Pelorus
apiVersion: charts.pelorus.dora-metrics.io/v1alpha1
metadata:
  name: pelorus-devspaces
spec:
  federate_openshift_monitoring:
    enabled: true
    metrics_filter:
      # pull devspaces metrics we're interested in
      - '{namespace=~".*-devspaces",pod=~"workspace.+"}'
  exporters:
    instances: []
  prometheus_storage: true
  prometheus_storage_pvc_capacity: 3Gi
  prometheus_storage_pvc_storageclass: gp2
  prometheus_retention_size: 2GB
achdmbp commented 4 months ago

@etsauer thank you. this has fixed the issue