beam-telemetry / cowboy_telemetry

Telemetry instrumentation for Cowboy
Apache License 2.0
29 stars 9 forks source link

adding cowboy_metrics_h after cowboy_telemetry_h causes crash #15

Open RoadRunnr opened 3 months ago

RoadRunnr commented 3 months ago

Setting up the cowboy stream handler like this:

#{metrics_callback => fun prometheus_cowboy2_instrumenter:observe/1,
   stream_handlers => [cowboy_telemetry_h, cowboy_metrics_h, cowboy_stream_h]

leads to a crash:

2024-03-25T12:25:21.637585+01:00 <0.3469.0> telemetry:execute/3 : error: Handler opentelemetry_cowboy_handlers has failed and has been detached. Class=error
Reason={case_clause,[]}
Stacktrace=[{otel_telemetry,pop_from_tracer_stack,1,
                [{file,
                     "/usr/src/erlang/erccn/_build/default/lib/opentelemetry_telemetry/src/otel_telemetry.erl"},
                 {line,110}]},
            {otel_telemetry,end_telemetry_span,2,
                [{file,
                     "/usr/src/erlang/erccn/_build/default/lib/opentelemetry_telemetry/src/otel_telemetry.erl"},
                 {line,49}]},
            {opentelemetry_cowboy,handle_event,4,
                [{file,
                     "/usr/src/erlang/erccn/_build/default/lib/opentelemetry_cowboy/src/opentelemetry_cowboy.erl"},
                 {line,79}]},
            {telemetry,'-execute/3-fun-0-',4,
                [{file,
                     "/usr/src/erlang/erccn/_build/default/lib/telemetry/src/telemetry.erl"},
                 {line,160}]},
            {lists,foreach_1,2,[{file,"lists.erl"},{line,1686}]},
            {cowboy_metrics_h,terminate,3,
                [{file,
                     "/usr/src/erlang/erccn/_build/default/lib/cowboy/src/cowboy_metrics_h.erl"},
                 {line,302}]},
            {cowboy_stream,terminate,3,
                [{file,
                     "/usr/src/erlang/erccn/_build/default/lib/cowboy/src/cowboy_stream.erl"},
                 {line,138}]},
            {cowboy_http2,terminate_stream_handler,4,
                [{file,
                     "/usr/src/erlang/erccn/_build/default/lib/cowboy/src/cowboy_http2.erl"},
                 {line,1334}]}]

the problem is that cowboy_telemetry_h abuses cowboy_metrics_h and overwrites the metrics_callback setting.

A workarround is to change the order of the handlers:

#{metrics_callback => fun prometheus_cowboy2_instrumenter:observe/1,
   stream_handlers => [cowboy_metrics_h, cowboy_telemetry_h, cowboy_stream_h]
josevalim commented 3 months ago

A pull request is welcome!