search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.82k stars 1.58k forks source link

Add runtime metrics collection to the docker source #1545

Open binarylogic opened 4 years ago

binarylogic commented 4 years ago

Docker has a runtime metrics feature. I'd like to collect this data, but I have a couple of important questions:

  1. How would this overlap with a server_metrics source? A source that collect host level CPU, memory, disk, and network metrics.
  2. Does it make sense to collect this data all in the same docker source, or should we rename the docker source to docker_logs so that we can add a new docker_metrics source?
LucioFranco commented 4 years ago

I think this should be integrated into a more general system level metrics collection source. My reasoning is because docker is basically a wrapper around cgroups which is a system level thing. Maybe we can find a way to expose the same metrics docker or no docker by just collecting general cgroup metrics.

jszwedko commented 3 years ago

Noting that a user requested this in Slack. Specifically the metrics exposed by http://api/containers/{container id or name}/stats Docker endpoint. They'd like to include the network stats as well.

gaby commented 9 months ago

Any updates on this? Can this be accomplished using host_metrics with cgroups?

jerrac commented 8 months ago

I've been wanting to give vector a try for a while now, but Docker metrics are essential.

I specifically need to be able to easily connect the metrics to the logs from the same container.

Would that be possible with the host metrics source? I'd think you'd need to know the cgroup for the container outputting the logs in order to connect the logs to the metrics if you relied on cgroups for the metric data. I don't see anything in the docs for docker logs about cgroups. So how would I connect the two?

I also don't see anything about cgroup specific network or disk IO in the host metrics docs. So if I needed that data at a container level, I'd be out of luck.

Which all leads me to also ask for a docker metrics source separate from everything else. Any chance we can get that sometime?

jszwedko commented 8 months ago

host_metrics does have support for reporting cgroup metrics now. This probably not the exact set of metrics available via http://api/containers/{container id or name}/stats though. I still think a dedicated source could make sense. We'd be happy to see a contribution if someone is motivated.