Erlang Cowboy OpenTelemetry Exporter didn't sent Traces to OTLP Collector Endpoint

[wdrdres3qew5ts21]

Hello everyone, I'm using Erlang with Opentelemetry Cowboy I can see Tracing Logs from Console already but it seem like when I switch to Exporters it didn't work. I had create example repository which already include my bugged source code. https://github.com/wdrdres3qew5ts21/erlang-opentelemetry-rebar3-bug Step to produces

1. Application will running on Port :8083
2. run Nginx at port :4317 to mock OTLP Collector
3. When Visit GET / Endpoint should sent Opentelemetry to Configured Endpoint
4. [Alternative Debugging] run docker-compose up then you should have real OTLP collector running at port gRPC:4317 too (I had already tried but it didn't work) you will not see any Trace in OTLP Collectors console logs

Expected Bahavior Erlang Application should sent Traces with :POST method to Nginx but currently it didn't work for both GRPC and HTTP

This command will give you successful connect with Open Telemetry

sys.config

[%% Kernel/logger
 {kernel, [{logger,[{handler,default,logger_std_h,#{}}]}
          %%,{logger_level,debug}
          ]},
   {opentelemetry,
  [{processors, 
    [{otel_batch_processor,
        #{exporter => {opentelemetry_exporter, #{protocol => http,
                                                 endpoints => ["http://localhost:4317"],
                                                 headers => [{"x-honeycomb-dataset", "experiments"}]}}}}]}]}
].

Dependency

[
{jsone,  "1.6.1"}, 
{cowboy, "2.7.0"},
{opentelemetry_api, "~> 1.0.0-rc.3"}, 
{opentelemetry, "1.0.0-rc.3"},
{hackney, ".*", {git, "git://github.com/benoitc/hackney.git", {branch, "master"}}},
{cowboy_telemetry, "0.4.0"},
{opentelemetry_cowboy, "0.1.0"},
{opentelemetry_exporter, "0.6.0"}
]

https://github.com/wdrdres3qew5ts21/erlang-opentelemetry-rebar3-bug

[kw7oe]

Hey @wdrdres3qew5ts21 can you try removing the protocol => http and see it if it works? As far as i know the currently supported protocols are: grpc, http_protobuf (Reference), http_json is still not implemented.

[wdrdres3qew5ts21]

@kw7oe I had tried many combination like gRPC and http already but it seem like it still didn't work. I'm using example from this Repositroy https://github.com/open-telemetry/opentelemetry-erlang/tree/main/apps/opentelemetry_exporter

[tsloughter]

Do you have any logs?

[tsloughter]

Nevermind, I may have seen the issue. I didn't get it completely working because your docker compose failed to run for me but changing the release to:

{relx, [{release, {dockerwatch, "1.0.0"}, [opentelemetry_exporter, dockerwatch, hackney,opentelemetry_cowboy]},

fixes an issue with needing the exporter application to boot before the SDK which uses it. Long story there that I'm trying to resolve.

But maybe that'll get you closer.

[wdrdres3qew5ts21]

@tsloughter It seem like after i change from otel_exporter_stdout to my current configure opentelemetry_exporter logs will not appear anymore. But I got some error logs like this instead OTLP tracer opentelemetry_exporter failed to initialize with exception error:{case_clause, http}

supakorn.t@cloudnative:~/ProjectCode/erlang-opentelemetry-rebar3-bug > rebar3 shell
===> Newly added dep opentelemetry_api is locked at a lower level. If you really want to unlock it, use 'rebar3 upgrade opentelemetry_api'
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling dockerwatch
Erlang/OTP 24 [erts-12.1.5] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [jit] [dtrace]

Eshell V12.1.5  (abort with ^G)
1> =WARNING REPORT==== 22-Dec-2021::19:38:02.501899 ===
OTLP tracer opentelemetry_exporter failed to initialize with exception error:{case_clause,
                                                                              http}
=======Before==============start_link==========> Booted unicode_util_compat
===> Booted idna
===> Booted mimerl
===> Booted certifi
===> Booted syntax_tools
===> Booted parse_trans
===> Booted ssl_verify_fun
===> Booted metrics
===> Booted hackney
===> Booted jsone
===> Booted cowlib
===> Booted ranch
===> Booted cowboy
===> Booted telemetry
===> Booted cowboy_telemetry
===> Booted opentelemetry_api
===> Booted opentelemetry
===> Booted telemetry_registry
===> Booted opentelemetry_cowboy
===> Booted opentelemetry_telemetry
===> Booted hpack
===> Booted chatterbox
===> Booted acceptor_pool
===> Booted gproc
===> Booted ctx
===> Booted grpcbox
===> Booted opentelemetry_exporter
===> Booted dockerwatch

[tsloughter]

Yea, that error about http is what @kw7oe mentioned, you need to use http_protobuf.

[kw7oe]

I managed to also look further into it. I think there are another two parts that require changes:

1. You might need to use grpc protocol instead because your otel-collector only expose port for grpc at 4317:

  otel-collector:
    image: ${OTELCOL_IMG}
    command: ["--config=/etc/otel-collector-config.yaml", "${OTELCOL_ARGS}"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "1888:1888"   # pprof extension
      - "8888:8888"   # Prometheus metrics exposed by the collector
      - "8889:8889"   # Prometheus exporter metrics
      - "13133:13133" # health_check extension
      - "4317:4137"        # OTLP gRPC receiver
      - "55670:55679" # zpages extension
    depends_on:
      - jaeger-all-in-one
      - zipkin-all-in-one

2. Else, you might want to try the following config format:

...
        #{exporter => {opentelemetry_exporter, #{
                                                 endpoints =>
                                                 %% instead of using string, use the tuple format
                                                 [{http, "localhost", 4137, []}]
                                                 }}}}]}]}
].

I can't manage to docker compose with the current docker-compose.yaml given, but I'm able to resolve it by fixing some of the issue above locally using the docker compose we have at: https://github.com/open-telemetry/opentelemetry-erlang-contrib/tree/main/examples/basic_elixir

Let me PR the changes I have on your repo there and maybe you can give it a try and see if it works?

[kw7oe]

@tsloughter i found out that when we configure the exporter endpoint with http with string format instead of tuple format, we will faced {:bad_scheme, "http"} error.

I have managed to replicated it by replacing the tuple format configuration with string in the opentelemetry-erlang-contrib example:

diff --git a/examples/basic_elixir/config/releases.exs b/examples/basic_elixir/config/releases.exs
index 96f4c2e..070d08a 100644
--- a/examples/basic_elixir/config/releases.exs
+++ b/examples/basic_elixir/config/releases.exs
@@ -8,5 +8,5 @@ config :opentelemetry,
          #
          # If you are running it locally, kindly change it to the correct
          # hostname such as `localhost`, `0.0.0.0` and etc.
-         exporter: {:opentelemetry_exporter, %{endpoints: [{:http, 'otel', 55681, []}]}}
+         exporter: {:opentelemetry_exporter, %{endpoints: ["http://localhost:55681"]}}
        }

and end up with the following logs:

~I think that's because we are expecting it in atom, and we didn't called atom_to_list when it's in uri_map format at here?~ <--- I'm wrong here

[wdrdres3qew5ts21]

@kw7oe @tsloughter Thank you so much it working now ! So Erlang Exporter Document need to have some updated README.md ? Because like you mention it support only tuple formatted.

[tsloughter]

Thanks @kw7oe, my guess is its just the part parsing the scheme. A string for endpoint should work, but likely a bug when parsing the endpoint.

[tsloughter]

Oh, and is this scehem failing issue on current main or rc3?

[kw7oe]

On rc3, let me also try out using main as well

[tsloughter]

Closing this because I think it was resolved, please reopen if this isn't the case.