Publish raw network metrics for components

jszwedko commented 2 years ago

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

It would be useful, in addition to the existing component_(received|sent)_bytes_total metric to see the raw bytes in / out of a component.

We should update the component spec to describe how / when these are emitted.

Attempted Solutions

No response

Proposal

Add the following metrics for all network-based components:

Counters:

component_network_received_bytes_total{component_id=...}
component_network_received_errors_total{component_id=...}
component_network_sent_bytes_total{component_id=...}
component_network_sent_errors_total{component_id=...}

Gauges:

component_network_connections{component_id=...,open=active|passive}

Components

Sources

[ ] Add raw network metrics for apache_metrics source
[ ] Add raw network metrics for aws_ecs_metrics source
[ ] Add raw network metrics for aws_kinesis_firehose source
[ ] Add raw network metrics for aws_s3 source
[ ] Add raw network metrics for aws_sqs source
[ ] #12769
[ ] Add raw network metrics for dnstap source
[ ] Add raw network metrics for docker_logs source
[ ] Add raw network metrics for eventstoredb_metrics source
[ ] Add raw network metrics for fluent source
[ ] Add raw network metrics for gcp_pubsub source
[ ] Add raw network metrics for heroku_logs source
[ ] Add raw network metrics for http source
[ ] Add raw network metrics for kafka source
[ ] Add raw network metrics for kubernetes_logs source
[ ] Add raw network metrics for logstash source
[ ] Add raw network metrics for mongodb_metrics source
[ ] Add raw network metrics for nats source
[ ] Add raw network metrics for nginx_metrics source
[ ] Add raw network metrics for postgresql_metrics source
[ ] Add raw network metrics for prometheus_remote_write source
[ ] Add raw network metrics for prometheus_scrape source
[ ] Add raw network metrics for redis source
[ ] Add raw network metrics for socket source
[ ] Add raw network metrics for splunk_hec source
[ ] Add raw network metrics for statsd source
[ ] Add raw network metrics for syslog source
[ ] Add raw network metrics for vector source

Some sources might only have input metrics or only output metrics.

Transforms

[ ] Add raw network metrics for aws_ec2_metadata transform

Sinks

[ ] Add raw network metrics for aws_cloudwatch_logs sink
[ ] Add raw network metrics for aws_cloudwatch_metrics sink
[ ] Add raw network metrics for aws_kinesis_firehose sink
[ ] Add raw network metrics for aws_kinesis_streams sink
[ ] Add raw network metrics for aws_s3 sink
[ ] Add raw network metrics for aws_sqs sink
[ ] Add raw network metrics for azure_blob sink
[ ] Add raw network metrics for azure_monitor_logs sink
[ ] Add raw network metrics for clickhouse sink
[ ] Add raw network metrics for datadog_archives sink
[ ] Add raw network metrics for datadog_events sink
[ ] Add raw network metrics for datadog_logs sink
[ ] #12770
[ ] Add raw network metrics for datadog_traces sink
[ ] Add raw network metrics for elasticsearch sink
[ ] Add raw network metrics for gcp_cloud_storage sink
[ ] Add raw network metrics for gcp_pubsub sink
[ ] Add raw network metrics for gcp_stackdriver_logs sink
[ ] Add raw network metrics for gcp_stackdriver_metrics sink
[ ] Add raw network metrics for honeycomb sink
[ ] Add raw network metrics for http sink
[ ] Add raw network metrics for humio_logs sink
[ ] Add raw network metrics for humio_metrics sink
[ ] Add raw network metrics for influxdb_logs sink
[ ] Add raw network metrics for influxdb_metrics sink
[ ] Add raw network metrics for kafka sink
[ ] Add raw network metrics for logdna sink
[ ] Add raw network metrics for loki sink
[ ] Add raw network metrics for nats sink
[ ] Add raw network metrics for new_relic sink
[ ] Add raw network metrics for new_relic_logs sink
[ ] Add raw network metrics for papertrail sink
[ ] Add raw network metrics for prometheus_exporter sink
[ ] Add raw network metrics for prometheus_remote_write sink
[ ] Add raw network metrics for pulsar sink
[ ] Add raw network metrics for redis sink
[ ] Add raw network metrics for sematext_logs sink
[ ] Add raw network metrics for sematext_metrics sink
[ ] Add raw network metrics for socket sink
[ ] Add raw network metrics for splunk_hec_logs sink
[ ] Add raw network metrics for splunk_hec_metrics sink
[ ] Add raw network metrics for statsd sink
[ ] Add raw network metrics for vector sink
[ ] Add raw network metrics for websocket sink

Some sinks might only have input metrics or only output metrics.

References

Version

vector 0.20.0 (x86_64-apple-darwin 2a706a3 2022-02-10)

binarylogic commented 2 years ago

I was thinking we could orient these around general network statistics, taking inspiration from the Prometheus node exporter network metrics. Something like:

network_received_bytes_total{component_id=...}
network_received_connections_total{component_id=...}
network_received_errors_total{component_id=...}
network_sent_bytes_total{component_id=...}
network_sent_connections_total{component_id=..., status=...}
network_sent_errors_total{component_id=..., status=...}
etc...

jszwedko commented 2 years ago

👍 I like that. I updated the issue to be generally about component network metrics.

binarylogic commented 2 years ago

Just noting, this is covered in the changes in https://github.com/vectordotdev/vector/pull/12912. We should be using the metrics noted there or update the spec first.

vectordotdev / vector