search

mozilla-services / heka

DEPRECATED: Data collection and processing made easy.
http://hekad.readthedocs.org/
Other
3.4k stars 531 forks source link

DockerLogInput and containers filter #1943

Open Oloremo opened 8 years ago

Oloremo commented 8 years ago

Hello!

Right now DockerLogInput plugin attaching to all running containers, sometimes it's not desirable. Docker API have a filter option for getting list of the containers and it will be really nice if DockerLogInput could use that filter, so we could filter out unneeded containers.

gjtempleton commented 8 years ago

This is something I've been thinking about implementing myself for both DockerLogs and DockerStats, borrowing from the message_matcher= syntax and allowing for wildcards.

Foreseeing allowing specification by container_matcher="ContainerLabel[foo] == 'bar'", ..."ContainerName == '*goldwasser'", ..."ContainerImage != 'nginx*'".

Is this something others would be interested in, and if so, does this seem a sensible way to proceed? I would think those three criteria would match every use case I can think of using current best practice?

Oloremo commented 8 years ago

Im totally ok with this example.

Oloremo commented 8 years ago

@gjtempleton Any updates? Just wondering...

fpytloun commented 8 years ago

Hello, this feature seems to be a must have to me :-) Otherwise it's not possible to pair decoders with specific containers.

gjtempleton commented 8 years ago

@Oloremo Sorry, life rather got in the way of this.

I wrote a quick and very dirty hack to get what I wanted out of this at work and only getting back around to writing something general enough I'd be happy enough to contribute back now. Probably halfway there with the logic.

@fpytloun If you're only wanting to send specific containers' output to specific decoders I think you could achieve what you want using fields_from_labels and using these fields with the message_matcher for each decoder.

fpytloun commented 8 years ago

@gjtempleton Thank you, that probably may work for my use-case. So I will have to use DockerInput with MultiDecoder and message_matcher for each decoder?