search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
18.13k stars 1.6k forks source link

New `opentelemetry` source and sink #1444

Open binarylogic opened 4 years ago

binarylogic commented 4 years ago

OpenTelemetry is a specification for collecting observability data.

Their collector and libraries are of questionable quality. We'd like Vector to support OT through their various protobufs and become the best OT collector.

We should break this down into smaller tasks, likely around their various data type (logs, metrics, and traces). I'd like to start with tracing, if possible, to introduce that type into our data model.

fzyzcjy commented 1 year ago

Hi, is there any updates? Thanks!

benjaminhuo commented 1 year ago

@spencergilbert I'm curious that now vector's OpenTelemetry Source can only ingest logs for now, metrics and traces are not yet supported?

spencergilbert commented 1 year ago

@spencergilbert I'm curious that now vector's OpenTelemetry Source can only ingest logs for now, metrics and traces are not yet supported?

Hi @benjaminhuo - metrics and traces are not yet supported, but we plan to support metrics in the near future.

srstrickland commented 1 year ago

Any updates or timeline here? Looking to build out an observability pipeline, and I was excited to use Vector for everything until I noticed the minimal support for OTel. An option we'll be evaluating now is building the entire observability pipeline with OTel collectors & exporters, since in theory they can handle logs, metrics, and traces (though the support for logs via OTel is still very young). My experimentation with vector for logs has been great so far, and it was a huge disappointment to find that we'll likely need another solution for traces (and maybe metrics). Here's the landscape as I see it:

For traces, there's exactly one source (datadog) and one sink (datadog).

For metrics, I am primarily interested in capturing (code-instrumented) application metrics. Ignoring the datadog libs/agent for a moment, it seems like my options for getting these into vector are:

  1. StatsD (ugh)
  2. Prometheus / scraping (not great for batch jobs)
  3. Prometheus / remote write
  4. Logs to metrics (feels like this should be used for edge cases, not the primary mechanism)

Publishing vector-shipped metrics to some endpoint are also a bit limited, but I could probably get by with what's available. The transformations for metrics seem very useful. In particular, the tag cardinality transform seems critical for maintaining downstream health (especially certain DBs). I have yet to find a similar OTel processor, so that's a big selling point for vector metrics, but I'm not sold on the mechanisms for getting metrics into vector. I would love to use OTel.

Am I missing something? Has anyone faced a similar situation? My options seem to be: vector for logs (and maybe metrics), OTel for tracing (and maybe metrics)... Or OTel for everything. (I am not sure about the OTel mechanism & data model for logs, though).

Reading through the history on this thread, it's hard for me to discount the likelihood that this work will never receive internal prioritization because it's counter to datadog's business model. I get it. Datadog needs to sell datadog. Maybe the answer here is for community contribution, as some folks here have already discussed. It would be great to get an idea on where things stand, so we're not just waiting for Godot.

cboudereau commented 1 year ago

Hi @srstrickland,

I had barely the same feelings for vector vs otel. Here is my comparison to help the vector team to enhance OTEL support.

You will find a comparison and working demo (end to end from signal to grafana dashboard for services and agents https://github.com/cboudereau/consul-demo/blob/main/service-discovery/up.sh). I made this demo during DataDog evaluation where I showed the solution to the DataDog Engineer solution from parsing with agent observability to monitor agents for vector and otelcol-contrib: https://github.com/cboudereau/consul-demo/tree/main/service-discovery/agents

In our case we choose both but we would like to reduce our complexity.

gaby commented 12 months ago

I also ran into the same issue. All this time I thought Vector supported OTEL, to find out it only supports part of it.

fpytloun commented 7 months ago

Any plan to increase priority for implementing full OTEL support?

hobbyhorse commented 1 month ago

This is the most depressing thing I have seen on github in a while.