search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.85k stars 1.58k forks source link

collectd metrics source #10123

Open archoversight opened 2 years ago

archoversight commented 2 years ago

Current Vector Version

vector 0.18.0 (x86_64-unknown-linux-gnu c77e085 2021-11-18)

Use-cases

We use collectd in our deployments alongside amazon-cloudwatch-agent and as we move to another hyperscaler we are looking to use vector across the board instead, however we need to collect the metrics from collectd and send them to a location of our choosing.

It would be great if vector would support a metric source that listened on a UDP port that collectd's network plugin could send data to.

The protocol is defined here: https://collectd.org/wiki/index.php/Binary_protocol

Attempted Solutions

We looked at replacing the collectd metrics with the ones that Vector does support, but Vector simply doesn't support as many, nor the plugins that we use.

Proposal

The proposal is to add a new metrics source that spins up a UDP port that collectd can connect to and send metrics directly to Vector for further processing/enrichment like any other metrics source.

References

ssgelm commented 1 year ago

This would be extremely useful for anyone trying to get GitHub Enterprise metrics into DataDog. Any chance it will get worked on?

jszwedko commented 1 year ago

This would be extremely useful for anyone trying to get GitHub Enterprise metrics into DataDog. Any chance it will get worked on?

Hey! This isn't something currently on our roadmap, but we'd happily to accept a contribution for it if anyone feels motivated 😄 The fluent is probably a good reference here since it is also a binary protocol.

jszwedko commented 1 year ago

Discussed this today and we think this could make sense as a encoding.codec option whereby it could be used with the socket source. We could still have a collectd source that is a wrapper.