teralytics / prometheus-ecs-discovery

A Prometheus discoverer that scrapes Amazon ECS and a generates file SD configuration file.
Apache License 2.0
257 stars 156 forks source link

Perform atomic writes for resulting service discovery YAML file #82

Open neilramsay opened 2 years ago

neilramsay commented 2 years ago

Perform atomic writes for resulting service discovery YAML file

We were seeing parsing error messages in Prometheus, as below:

ts=2022-02-08T17:59:59.920Z caller=file.go:337 level=error component="discovery manager scrape" discovery=file msg="Error reading file" path=/etc/prometheus/prometheus_ecs_container_endpoints.123456789.yml err="yaml: line 186: could not find expected ':'"

On manual inspection of the files, they appeared complete and could be parsed. This indicates that the file is not complete at the time Prometheus is reading it.

The https://github.com/natefinch/atomic library allows writing the service discovery file, and then atomically moving it in place.