Use docker events instead of periodic scanning of running docker containers

[Kamilcuk]

Is your feature request related to a problem? Please describe.

The problem is that scanning docker containers misses short-lived docker containers that live shorter than configured timeout.

Describe the solution you'd like

I would want promtail to listen on docker events, and when it detects an event that a docker container is started, query it's logs. That would be beneficial in every aspect, reduce cpu usage from periodic docker scanning.

docker events
2022-12-15T16:20:26.595520362-05:00 container start 54963028d283904326f0e8b4cfb525b1de1be39bc5faf96312d52830d7c19cec (image=alpine, name=infallible_jones)

Describe alternatives you've considered

I did not consider any alternatives.

Additional context

Receiving asynchronous notifications generally is better than periodic polling.

[jeschkies]

Thanks for figuring this out. I'd like that but we are using the service discovery from Prometheus for Docker (https://github.com/prometheus/prometheus/blob/main/discovery/moby/docker.go#L176). It polls. We'd have to patch Prometheus and use events instead.

[Kamilcuk]

Hi, thanks. What can I do, to make absolutely sure that promtail collects the logs?

Assuming docker_sd_configs: rerfesh_interval: "2s" and I run a simple docker container docker run -ti --rm alpine sh -c 'sleep $1; echo hello world; sleep $2", what should I do to make absolutely completely sure that hello world will be collected by promtail?

• should I wait at least 2 seconds before running my command?
• should I wait at least 2 seconds after running my command?
• should I wait at least 2 seconds before and after running my command?
• should I wait at least 3 seconds? 4 seconds?

Hope you can clarify it for me, thanks.

The issue is that promtail is missing a lot of containers. With refresh_interval: 2 I have to wait at least 4 seconds before and after the command to make sure it is collected.