search

NOAA-GSL / VxIngest

Other
2 stars 0 forks source link

metrics name is no longer consistent per run because it includes the container hostname #324

Closed randytpierce closed 6 months ago

randytpierce commented 6 months ago

A problem with the scraper happened because the metric name was including the hostname where the ingest runs. Since we moved the ingest into a container the hostnames are no longer consistent. The textfile collector doesn't remove the metrics file, it just scans it. That means it is expected to be the same name each time and the contents just overwritten. Because the hostname was included in the metrics file name and the container host name changes the metrics files were just piling up. Eventually this caused the scraper to fail dramatically.

randytpierce commented 6 months ago

This is working now. The latest problem was that the metrics name used to include the hostname but when we are running from a temporary container the host name always changes. This caused the metrics .prom files to be uniq and it overwhelmed the textfile collector that the node exporter uses to collect the metrics. I removed the hostname part of the metrics name.