grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.91k stars 3.45k forks source link

Promtail should abandon non-existent files #3716

Closed boniek83 closed 3 years ago

boniek83 commented 3 years ago

Describe the bug

level=error ts=2021-05-12T09:38:46.447252922Z caller=filetarget.go:276 msg="failed to tail file, stat failed" error="stat /var/log/pods/seldon-analytics_seldon-analytics-prometheus-node-exporter-c7xmn_bc61dab4-450f-4540-a145-1ed57ec9bfa5/prometheus-node-exporter/2.log: no such file or directory" filename=/var/log/pods/seldon-analytics_seldon-analytics-prometheus-node-exporter-c7xmn_bc61dab4-450f-4540-a145-1ed57ec9bfa5/prometheus-node-exporter/2.log

I'm seeing errors like these in promtail output. If file doesn't exist because promtail was down for extended amount of time then just log the fact and drop it from tracked files and don't remind me about it every 10 seconds forever.

Something like this also polutes error logs: level=error ts=2021-05-12T08:49:29.705845785Z caller=filetarget.go:264 msg=" failed to remove directory from watcher" error="can't remove non-existent inotify watch for: /var/log/pods/polyaxon_plx-bb3f99e9354642d0b20c17301b50b62c-master-0_c26e02f1-425c-447b-8944-23816ea18873/polyaxon-init-job" Is this really an error? What can I do about it?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

andrejshapal commented 1 year ago

Not stale