techno-tim / launchpad

A collection of quick starters for ansible, kubernetes, docker, linux, windows, and more. Great for HomeLabs!
https://technotim.live
1.47k stars 478 forks source link

Prometheus yaml parsing error: "parsing YAML file /etc/prometheus/config_out/prometheus.env.yaml: empty duration string" #46

Closed MarkhamLee closed 3 months ago

MarkhamLee commented 11 months ago

I ran into this issue while attempting to deploy kube-prometheus-stack as described here: https://www.youtube.com/watch?v=fzny5uUaAeY&t=1210s,

The specific error I got was:msg="Error loading config (--config.file=/etc/prometheus/config_out/prometheus.env.yaml)" file=/etc/prometheus/config_out/prometheus.env.yaml err="parsing YAML file /etc/prometheus/config_out/prometheus.env.yaml: empty duration string"

Per this: https://github.com/prometheus-operator/prometheus-operator/issues/5197 the issue comes from needing to update the Prometheus operator CRDs.

Basically, the scrape and evaluation intervals are no longer defined as default values so you need to add them, I resolved it by adding the following to the Prometheus section of the values.yaml file: scrapeInterval: 30s evaluationInterval: 30s

I.e. the easy solution seems to be update the values.yaml, is to pull the latest CRDs in the values.yaml

byrn-baker commented 6 months ago

I tried this in the values.yml and I am still getting the same error. ts=2024-04-26T15:09:27.765Z caller=main.go:521 level=error msg="Error loading config (--config.file=/etc/prometheus/config_out/prometheus.env.yaml)" file=/etc/prometheus/config_out/prometheus.env.yaml err="parsing YAML file /etc/prometheus/config_out/prometheus.env.yaml: empty duration string"

Also noticing that the alertmanager pods are unable to start as well. Error from server (BadRequest): container "alertmanager" in pod "alertmanager-prometheus-alertmanager-0" is waiting to start: PodInitializing it maybe waiting for prometheus-prometheus-prometheus-0 pod though.

MarkhamLee commented 6 months ago

I would double-check your formatting, if you're getting that error it sounds like you either updated the wrong field or its an indentation problem.

MichaelKora commented 5 months ago

hey @byrn-baker ..i sthis issue still open?..i had this issue few days ago...and the issue was that indentation for some configs i added wasnt correct...compare with the official values.yaml and make sure that any data you try to change is well link.

Eg. i was trying to update tolerations and was trying to update prometheus.talorations but the correct way was to update prometheus.prometheusSpec.tolerations...

so i will just make sure to verify that every config i change is well positioned