search

open-telemetry / opentelemetry-js

OpenTelemetry JavaScript Client
https://opentelemetry.io
Apache License 2.0
2.72k stars 792 forks source link

opentelemetry/exporter-metrics-otlp-http Does not send metrics to collector (with last collector version 0.40.0) #2675

Closed Nighttraveler closed 2 years ago

Nighttraveler commented 2 years ago

Hello guys, We wanted to use https://github.com/GoogleChrome/web-vitals#browser-support to get some metrics from our browser instrumentation and send them through opentelemetry-js. In order to accomplish that we started to make some tests with the examples provided here.

What version of OpenTelemetry are you using?

I'm using 0.27.0 opentelemetry-js version (this repo in main branch) and otelcontribcol 0.40.0

What version of Node are you using?

v12.16.3

Please provide the code you used to setup the OpenTelemetry SDK

At the moment the exporter does not work with the examples provided. Just changed this in the example (tracer-web/metrics/index.js) const labels = { pid: 1, environment: 'staging' }; because 'process' is undefined

What did you do?

We just simply start up the 'tracer-web' examples. All the traces examples work as expected and arrive to our collector, but metrics doesn't.

If possible, provide a recipe for reproducing the error.

What did you expect to see?

I expected to see the metrics in our collector/backed, like all the other traces examples that works correctly.

What did you see instead?

In Network tab from the browser I have a 400 Bad request response from the collector.

{"code":3,"message":"unknown field \"doubleSum\" in v1.Metric"}

Additional context

* Collector yml config file:

receivers: otlp: protocols: grpc: http: cors_allowed_origins:

Hope the information is enough to reproduce the issue, if not, please reach me and I provide more details.

morigs commented 2 years ago

Same issue here. Also tested with otelcol 0.41.0, otelcontribcol 0.40.0 and otelcontribcol 0.41.0

morigs commented 2 years ago

I believe this is due to inconsistencies between payload of the Metrics and actual proto type.

This is caused by this change. This changed in proto 0.9.0 which delivered to the collector in version 0.33.0.

Perhaps fix is quite easy: just rename fields to the new scheme and remove unused (int) fields.

dyladan commented 2 years ago

Yeah our proto is a little out of date. i'll update it

morigs commented 2 years ago

Looking forward! If any help is needed, just let me know

dyladan commented 2 years ago

I won't have time to tackle this until monday so if you want to do it before then I will try to keep an eye out for notifications and merge/release the fix over the weekend. If you can wait until monday i'll just handle it then.

morigs commented 2 years ago

@dyladan sorry for pinging you, is there any success?

RomeroGaliza commented 2 years ago

Same issue here!

morigs commented 2 years ago

@dyladan sorry for disturbing you again. Could you pls add help-needed badge or something. Just to draw attention of possible contributors to the issue. Or maybe you can provide a high-level list of steps required for proto upgrade, so I can do it myself? That would be awesome

dyladan commented 2 years ago

In order to get this working, the proto transformations need to be updated. The transformations are in experimental/packages/opentelemetry-exporter-metrics-otlp-http/src/transformMetrics.ts and depend on the types in packages/exporter-trace-otlp-http/src/types.ts. Because of this unfortunate coupling, the proto files, their types (hand rolled), and the transformations (also hand rolled) need to be updated for all 6 otlp exporters at the same time. If you want to get the fix out ASAP, I would recommend you do that.

Alternatively, I am working on a separate package which will handle transformations into the intermediate OTLP JSON format and serialization as protobuf in #2691. Once that package is completed, the metrics exporters will be refactored to use it (and it already has updated proto). Depending on how time-sensitive this is for you, it may be prudent to just wait for that (i would estimate a couple weeks).

mbecca commented 2 years ago

Hello, is there any progress?

sgracias1 commented 2 years ago

I ran a test with the OTLP trace exporter and noticed it fails to connect to the collector for version > 0.42.0. Does this issue track the trace exporter as well, or should that be a separate issue?

hejkerooo commented 2 years ago

"message":"Bad Request","name":"OTLPExporterError","data":"{\"code\":3,\"message\":\"unknown field \\\"doubleSum\\\" in v1.Metric\"}","code":"400"}

This is the error I'm receiving when using metrics exporter

  otlp:
    protocols:
      grpc:
      http:
  zipkin:

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
  logging:
  zipkin:
    endpoint: "http://zipkin:9411/api/v2/spans"

processors:
  batch:

extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp, zipkin]
      exporters: [zipkin, jaeger, logging]
      processors: [batch]
    metrics:
      receivers: [ otlp ]
      exporters: [ logging ]
const metricExporter = new OTLPMetricExporter({
  url: 'http://localhost:4318/v1/metrics',
});
radoslavirha commented 2 years ago

It's still an issue with "@opentelemetry/exporter-metrics-otlp-http": "0.28.0" and otel/opentelemetry-collector-contrib:0.51.0. Any ideas how to fix it? Thanks

gvelez17 commented 2 years ago

Using same versions here (0.28.0 and 0.51.0) and also not seeing metrics appearing in the collector

jeevanearform commented 2 years ago

Same issue here. I am on @opentelemetry/exporter-metrics-otlp-http:0.28.0 and using docker image otel/opentelemetry-collector:latest

mplachter commented 2 years ago

Same issue here. I thought the latests 0.28.0 was going to have a newer otlp metric proto but it does not and still will not work with anything passed 0.42.0

mplachter commented 2 years ago

It's still an issue with "@opentelemetry/exporter-metrics-otlp-http": "0.28.0" and otel/opentelemetry-collector-contrib:0.51.0. Any ideas how to fix it? Thanks

gotta use 0.42.0 or older it seems.

radoslavirha commented 2 years ago

I had to use 0.29.0 docker image .... I had some trouble with <=0.42.0 too

mplachter commented 2 years ago

@dyladan any progress on this?

dyladan commented 2 years ago

I'm sorry i forgot to create a release but 1.3.0/0.29.0 was released at the end of last week https://github.com/open-telemetry/opentelemetry-js/pull/2995 the most recent version should work

mplachter commented 2 years ago

@dyladan trying both 0.29.0 and 0.29.1 and i'm getting these grpc errors flooding the collector with no traces or metrics.

2022-06-02T16:19:46.988Z        warn    zapgrpc/zapgrpc.go:191  [core] grpc: Server.Serve failed to create ServerTransport:  connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"\\x16\\x03\\x01\\x01o\\x01\\x00\\x01k\\x03\\x03\\xca\\xd0\\v\\xfd\\xc8\\u007f)\\x8av\\xd3\\x11\\xcci\""        {"grpc_log": true}
2022-06-02T16:19:50.642Z        warn    zapgrpc/zapgrpc.go:191  [core] grpc: Server.Serve failed to create ServerTransport:  connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"\\x16\\x03\\x01\\x01o\\x01\\x00\\x01k\\x03\\x03\\x06[u\\x00\\b8/\\x16aM;\\xec\\x92\""  {"grpc_log": true}
2022-06-02T16:19:50.642Z        warn    zapgrpc/zapgrpc.go:191  [core] grpc: Server.Serve failed to create ServerTransport:  connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"\\x16\\x03\\x01\\x01o\\x01\\x00\\x01k\\x03\\x03\\xf5\\x15\\xb6S\\xef\\x1aZ\\x15|\\t\\x05\\xd6{\""

Tired with 0.42.0, 0.44.0 and 0.52.0 otel collector contribs

dyladan commented 2 years ago

That looks like a collector issue not an issue with OTel js

dyladan commented 2 years ago

@pichlermarc if you have time can you look into this?

mplachter commented 2 years ago

I'm running this locally in a docker compose.. Not in a deployed env.

I'll double check again to make sure nothing is trying to communicate with otel over the grpc port.

mplachter commented 2 years ago

Yeah I did double check and also i get this from the otel-js side.

code":14,"details":"No connection established","metadata":{}},"exception":true,"hostname":"machine_name","level":"error","message":"uncaughtException: 14 UNAVAILABLE: No connection established\nError: 14 UNAVAILABLE: No connection established\n    at Object.callErrorFromStatus (/..../@grpc/grpc-js/src/call.ts:81:24)
mplachter commented 2 years ago

The Go modules work no problem with the collectors only getting this behavior using the js packages.

pichlermarc commented 2 years ago

@mplachter you're using @opentelemetry/exporter-metrics-otlp-grpc, right? :thinking:

I looked into your issue and found that the @opentelemetry/exporter-metrics-otlp-http works, but @opentelemetry/exporter-metrics-otlp-grpc is currently broken. It seems like a PR after #2886 introduced another bug. This causes the exporter to try and open a secure connection on an insecure server.

I'm currently working on a fix.

Edit: the fix should be ready over at #3019 :slightly_smiling_face:

dyladan commented 2 years ago

@mplachter @pichlermarc can this be closed?

pichlermarc commented 2 years ago

@dyladan for me it is working again. But we have not released that fix yet.

lamroger commented 1 year ago

Running into this again on 0.35.1. I'm using RemixInstrumentation. Maybe that is sending the wrong shape?

pichlermarc commented 1 year ago

Hi @lamroger, thanks for reaching out. Could you please open a new bug for the problem you're seeing?

It has gotten a bit hard to follow the thread here, and a lot has changed since this issue here was first opened.

Pratap22 commented 1 year ago

I keep getting this error.

{"stack":"OTLPExporterError: Not Found\n    at IncomingMessage.<anonymous> (/Users/admin/personal/otel/otel-collector-sender/node_modules/@opentelemetry/otlp-exporter-base/build/src/platform/node/util.js:103:39)\n    at IncomingMessage.emit (node:events:402:35)\n    at endReadableNT (node:internal/streams/readable:1343:12)\n    at processTicksAndRejections (node:internal/process/task_queues:83:21)","message":"Not Found","name":"OTLPExporterError","data":"404 page not found\n","code":"404"}

package.json

{
"dependencies": {
    "@opentelemetry/api": "^1.4.1",
    "@opentelemetry/sdk-node": "^0.41.2",
    "opentelemetry-instrumentation-express": "^0.39.1"
  }

}

otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:8000
      http:
        endpoint: 0.0.0.0:4318

exporters:
  prometheus:
    endpoint: collector:7777
    namespace: mysearcher

extensions:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
  telemetry:
    logs:
      level: info

version: '3.5'
services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    ports:
      - 9090:9090
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus_data:/prometheus
  collector:
    image: otel/opentelemetry-collector-contrib
    container_name: collector
    command: [--config=/etc/otel-collector-config.yaml]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - 8000:8000
      - "4318:4318"
  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - 3000:3000

Can you help me why am I getting the error?

pj-pi commented 8 months ago

Hi @Pratap22 , I got the same error as yours. Did you manage to solve the issue?