VictoriaMetrics / VictoriaMetrics

VictoriaMetrics: fast, cost-effective monitoring solution and time series database
https://victoriametrics.com/
Apache License 2.0
11.27k stars 1.13k forks source link

Adds firehose http endpoint protocol integration for opentelemetry ingestion. #6037

Open f41gh7 opened 3 months ago

f41gh7 commented 3 months ago

Is your feature request related to a problem? Please describe

It's possible to forward AWS CloudWatch metrics to the VictoriaMetrics with Firehose http endpoint opentelemetry format. https://aws.amazon.com/blogs/aws/cloudwatch-metric-streams-send-aws-metrics-to-partners-and-to-your-apps-in-real-time/

But it has the following issues:

Describe the solution you'd like

Describe alternatives you've considered

No response

Additional information

No response

valyala commented 2 months ago

FYI, the compatibility with AWS Firehose should be fixed in the commit 509df44d03631a863fab00e202a6449c19d784e1

valyala commented 2 months ago

FYI, the -opentelemetry.usePrometheusNaming flag has been introduced in the commit fb42380ef374c394053b4cc4f2b083ed4afc7d09 . If this flag is set, then VictoriaMetrics automatically converts metric names and label names into Prometheus-compatible names for samples ingested via OpenTelemetry format.

valyala commented 2 months ago

FYI, vmagent and single-node VictoriaMetrics properly return responses to Amazon Firehose starting from v1.100.0. They also support the -opentelemetry.usePrometheusNaming command-line flag starting from v1.100.0 release.

AndrewChubatiuk commented 2 months ago

@f41gh7 we're converting otel metrics in the same manner as it's done in collector, they are also storing array and map attributes as JSON