search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.37k stars 1.51k forks source link

vector_processed_events_total metric missing from HTTP sink #6069

Closed hreidar closed 3 years ago

hreidar commented 3 years ago

Vector Version

vector:0.11.1-debian

Vector Configuration File

# metrics
[sources.p2-metrics]
type = "internal_metrics"

### I have several file sources and add_fields transforms here which I have omitted from this example  ###

## sinks
[sinks.to_dedup]
  type = "http"
#  inputs = ["sampler"]
  inputs = ["back01-oss-fields",
            "cnr01-ipv6-fields",
            "cnr02-ipv6-fields",
            "cnr01-ipv4-fields",
            "cnr02-ipv4-fields",
            "cnr01-autod-fields",
            "cnr02-autod-fields"
           ]
  uri = "http://127.0.0.1:9001"
  encoding.codec = "json"
  healthcheck = false
  buffer.type = "disk"
  buffer.max_size = 104900000
  request.concurrency = "adaptive"

[sinks.p2-to-prometheus]
  type = "prometheus_exporter"
  inputs = ["p2-metrics"]
  address = "0.0.0.0:9501"
  default_namespace = "p2-ingress1"
  buckets = [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
  quantiles = [0.5, 0.75, 0.9, 0.95, 0.99]

Debug Output

Debug output: https://gist.github.com/hreidar/dcb9506c8a0a0501110991d9827df32a Metrics output: https://gist.github.com/hreidar/29889bca10c9aa2516038a191e32ba5f

Expected Behavior

I should have vector_processed_events_total for the HTTP sink.

Actual Behavior

vector_processed_events_total for HTTP sink not in my metrics output.

Example Data

Additional Context

The example I have here is from a Vector container running on a VM. It is controlled by docker-compose. I also have a similar container that is running in a pod on Kubernetes. I see the same behavior there, no vector_processed_events_total for HTTP sink.

References

hreidar commented 3 years ago

Hi, I also notice that some of the "vector_adaptiveconcurrency" metrics are missing labels that I would expect to be there. For example this one: vector_http_client_response_rtt_nsbucket{le="+Inf",status="200 OK"} has no "component" labels

jszwedko commented 3 years ago

I'm noting that the metrics do appear on master as of 59b609073 so it's possible this was fixed.

I also note that they never update which seems related to https://github.com/timberio/vector/issues/5916

hreidar commented 3 years ago

Just tryed the nightly and I get an error using the config I had working with 0.11.1

ERROR vector::cli: Configuration error: unknown variant `prometheus_exporter`, expected one of `aws_cloudwatch_logs`, `aws_cloudwatch_metrics`, `aws_kinesis_firehose`, `aws_kinesis_streams`, `aws_s3`, `azure_monitor_logs`, `blackhole`, `clickhouse`, `console`, `datadog_logs`, `datadog_metrics`, `elasticsearch`, `file`, `gcp_cloud_storage`, `gcp_pubsub`, `gcp_stackdriver_logs`, `honeycomb`, `http`, `humio_logs`, `influxdb_logs`, `influxdb_metrics`, `kafka`, `logdna`, `loki`, `new_relic_logs`, `papertrail`, `prometheus`, `pulsar`, `sematext`, `socket`, `splunk_hec`, `statsd`, `vector` for key `sinks.p2-to-prometheus`
jszwedko commented 3 years ago

@hreidar Can you post the output of vector --version? I'm not able to reproduce that. Also, how did you install the nightly? I'm wondering if it pulled an old version.

hreidar commented 3 years ago 
root@log02:/opt/vector# docker ps
CONTAINER ID        IMAGE                            COMMAND             CREATED             STATUS                                   PORTS               NAMES
a3710d1b670f        timberio/vector:nightly-debian   "/usr/bin/vector"   5 seconds ago       Restarting (78) Less than a second ago                       vector_ingress1_1
5422838c8a6e        timberio/vector:0.11.1-debian    "/usr/bin/vector"   4 hours ago         Up 4 hours                                                   vector_ingress2_1
62bf4274082b        timberio/vector:0.11.1-debian    "/usr/bin/vector"   4 hours ago         Up 4 hours                                                   vector_output_1
root@log02:/opt/vector# 
root@log02:/opt/vector# docker logs -f vector_ingress1_1 
Jan 15 15:13:17.071 ERROR vector::cli: Configuration error: unknown variant `prometheus_exporter`, expected one of `aws_cloudwatch_logs`, `aws_cloudwatch_metrics`, `aws_kinesis_firehose`, `aws_kinesis_streams`, `aws_s3`, `azure_monitor_logs`, `blackhole`, `clickhouse`, `console`, `datadog_logs`, `datadog_metrics`, `elasticsearch`, `file`, `gcp_cloud_storage`, `gcp_pubsub`, `gcp_stackdriver_logs`, `honeycomb`, `http`, `humio_logs`, `influxdb_logs`, `influxdb_metrics`, `kafka`, `logdna`, `loki`, `new_relic_logs`, `papertrail`, `prometheus`, `pulsar`, `sematext`, `socket`, `splunk_hec`, `statsd`, `vector` for key `sinks.p2-to-prometheus`
Jan 15 15:13:18.377 ERROR vector::cli: Configuration error: unknown variant `prometheus_exporter`, expected one of `aws_cloudwatch_logs`, `aws_cloudwatch_metrics`, `aws_kinesis_firehose`, `aws_kinesis_streams`, `aws_s3`, `azure_monitor_logs`, `blackhole`, `clickhouse`, `console`, `datadog_logs`, `datadog_metrics`, `elasticsearch`, `file`, `gcp_cloud_storage`, `gcp_pubsub`, `gcp_stackdriver_logs`, `honeycomb`, `http`, `humio_logs`, `influxdb_logs`, `influxdb_metrics`, `kafka`, `logdna`, `loki`, `new_relic_logs`, `papertrail`, `prometheus`, `pulsar`, `sematext`, `socket`, `splunk_hec`, `statsd`, `vector` for key `sinks.p2-to-prometheus`
Jan 15 15:13:19.620 ERROR vector::cli: Configuration error: unknown variant `prometheus_exporter`, expected one of `aws_cloudwatch_logs`, `aws_cloudwatch_metrics`, `aws_kinesis_firehose`, `aws_kinesis_streams`, `aws_s3`, `azure_monitor_logs`, `blackhole`, `clickhouse`, `console`, `datadog_logs`, `datadog_metrics`, `elasticsearch`, `file`, `gcp_cloud_storage`, `gcp_pubsub`, `gcp_stackdriver_logs`, `honeycomb`, `http`, `humio_logs`, `influxdb_logs`, `influxdb_metrics`, `kafka`, `logdna`, `loki`, `new_relic_logs`, `papertrail`, `prometheus`, `pulsar`, `sematext`, `socket`, `splunk_hec`, `statsd`, `vector` for key `sinks.p2-to-prometheus`
Jan 15 15:13:20.983 ERROR vector::cli: Configuration error: unknown variant `prometheus_exporter`, expected one of `aws_cloudwatch_logs`, `aws_cloudwatch_metrics`, `aws_kinesis_firehose`, `aws_kinesis_streams`, `aws_s3`, `azure_monitor_logs`, `blackhole`, `clickhouse`, `console`, `datadog_logs`, `datadog_metrics`, `elasticsearch`, `file`, `gcp_cloud_storage`, `gcp_pubsub`, `gcp_stackdriver_logs`, `honeycomb`, `http`, `humio_logs`, `influxdb_logs`, `influxdb_metrics`, `kafka`, `logdna`, `loki`, `new_relic_logs`, `papertrail`, `prometheus`, `pulsar`, `sematext`, `socket`, `splunk_hec`, `statsd`, `vector` for key `sinks.p2-to-prometheus`
Jan 15 15:13:22.747 ERROR vector::cli: Configuration error: unknown variant `prometheus_exporter`, expected one of `aws_cloudwatch_logs`, `aws_cloudwatch_metrics`, `aws_kinesis_firehose`, `aws_kinesis_streams`, `aws_s3`, `azure_monitor_logs`, `blackhole`, `clickhouse`, `console`, `datadog_logs`, `datadog_metrics`, `elasticsearch`, `file`, `gcp_cloud_storage`, `gcp_pubsub`, `gcp_stackdriver_logs`, `honeycomb`, `http`, `humio_logs`, `influxdb_logs`, `influxdb_metrics`, `kafka`, `logdna`, `loki`, `new_relic_logs`, `papertrail`, `prometheus`, `pulsar`, `sematext`, `socket`, `splunk_hec`, `statsd`, `vector` for key `sinks.p2-to-prometheus`
Jan 15 15:13:25.328 ERROR vector::cli: Configuration error: unknown variant `prometheus_exporter`, expected one of `aws_cloudwatch_logs`, `aws_cloudwatch_metrics`, `aws_kinesis_firehose`, `aws_kinesis_streams`, `aws_s3`, `azure_monitor_logs`, `blackhole`, `clickhouse`, `console`, `datadog_logs`, `datadog_metrics`, `elasticsearch`, `file`, `gcp_cloud_storage`, `gcp_pubsub`, `gcp_stackdriver_logs`, `honeycomb`, `http`, `humio_logs`, `influxdb_logs`, `influxdb_metrics`, `kafka`, `logdna`, `loki`, `new_relic_logs`, `papertrail`, `prometheus`, `pulsar`, `sematext`, `socket`, `splunk_hec`, `statsd`, `vector` for key `sinks.p2-to-prometheus`
Jan 15 15:13:29.679 ERROR vector::cli: Configuration error: unknown variant `prometheus_exporter`, expected one of `aws_cloudwatch_logs`, `aws_cloudwatch_metrics`, `aws_kinesis_firehose`, `aws_kinesis_streams`, `aws_s3`, `azure_monitor_logs`, `blackhole`, `clickhouse`, `console`, `datadog_logs`, `datadog_metrics`, `elasticsearch`, `file`, `gcp_cloud_storage`, `gcp_pubsub`, `gcp_stackdriver_logs`, `honeycomb`, `http`, `humio_logs`, `influxdb_logs`, `influxdb_metrics`, `kafka`, `logdna`, `loki`, `new_relic_logs`, `papertrail`, `prometheus`, `pulsar`, `sematext`, `socket`, `splunk_hec`, `statsd`, `vector` for key `sinks.p2-to-prometheus`
root@log02:/opt/vector#
jamtur01 commented 3 years ago

@bruceg Looks like something is not okay with the prometheus to prometheus_exporter sink alias/rename?

hreidar commented 3 years ago

it seems so, tried to change it to "prometheus" and then I'm getting this error:

ERROR vector::cli: Configuration error: unknown field `default_namespace`, expected one of `namespace`, `address`, `buckets`, `flush_period_secs` for key `sinks.p2-to-prometheus`
jszwedko commented 3 years ago

@hreidar I think you are running an older version of vector. Can you try to docker pull?

I see:

➜  /tmp docker run -v /tmp/tmp.toml:/tmp/tmp.toml timberio/vector:nightly-debian --config /tmp/tmp.toml
Jan 15 16:02:19.745  INFO vector::app: Log level is enabled. level="vector=info,codec=info,file_source=info,tower_limit=trace,rdkafka=info"
Jan 15 16:02:19.745  INFO vector::sources::host_metrics: PROCFS_ROOT is unset. Using default '/proc' for procfs root.
Jan 15 16:02:19.745  INFO vector::sources::host_metrics: SYSFS_ROOT is unset. Using default '/sys' for sysfs root.
Jan 15 16:02:19.745  INFO vector::app: Loading configs. path=[("/tmp/tmp.toml", None)]
Jan 15 16:02:19.752  INFO vector::sources::stdin: Capturing STDIN.
Jan 15 16:02:19.755  INFO vector::topology: Running healthchecks.
Jan 15 16:02:19.756  INFO vector::topology: Starting source. name="metrics"
Jan 15 16:02:19.756  INFO vector::topology: Starting source. name="in"
Jan 15 16:02:19.756  INFO vector::topology: Starting sink. name="out"
Jan 15 16:02:19.756  INFO vector::topology: Starting sink. name="prometheus"
Jan 15 16:02:19.756  INFO vector: Vector has started. version="0.12.0" git_version="v0.11.0-336-g7fa399f" released="Fri, 15 Jan 2021 05:16:18 +0000" arch="x86_64"
Jan 15 16:02:19.756  INFO vector::topology::builder: Healthcheck: Disabled.
Jan 15 16:02:19.756  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
Jan 15 16:02:19.756  INFO vector::topology::builder: Healthcheck: Passed.
Jan 15 16:02:19.756  INFO source{component_kind="source" component_name=in component_type=stdin}: vector::sources::stdin: Finished sending.
^CJan 15 16:02:20.680  INFO vector: Vector has stopped.
➜  /tmp cat /tmp/tmp.toml                                                                              
[sources.in]
type = "stdin"

[sinks.out]
  type = "http"
  inputs = ["in"]
  uri = "http://127.0.0.1:9001"
  encoding.codec = "json"
  healthcheck = false
  request.concurrency = "adaptive"

[sources.metrics]
type = "internal_metrics"

[sinks.prometheus]
type = "prometheus_exporter"
inputs = ["metrics"]
address = "127.0.0.1:10000"
hreidar commented 3 years ago

You are absolutely right :-)

docker images ls
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
root@log02:/opt/vector# docker images
REPOSITORY                         TAG                 IMAGE ID            CREATED             SIZE
timberio/vector                    0.11.1-debian       b0fd270dc6ab        4 weeks ago         212MB
timberio/vector                    0.11.0-debian       6241ce5fe305        6 weeks ago         212MB
timberio/vector                    nightly-debian      57bcab2c5076        3 months ago        160MB
docker.elastic.co/beats/filebeat   7.9.2               38a0be006475        3 months ago        449MB
timberio/vector                    0.10.0-debian       61cb60811ee4        5 months ago        145MB
hreidar commented 3 years ago

my bad :-/

jszwedko commented 3 years ago

No worries! I've been bitten by stale docker image tags too :smile:

hreidar commented 3 years ago

It is up and running now and I'm seeing the vector_processed_events_total for HTTP sink.

vector_processed_events_total{component_kind="sink",component_name="to_dedup",component_type="http"} 24 1610728157995
hreidar commented 3 years ago

so should I just close this issue?

jszwedko commented 3 years ago

Yeah, let's close this since the stale prometheus output issue is covered by https://github.com/timberio/vector/issues/5916

hreidar commented 3 years ago

ok, thanks so much for your time and info