obscure config error

[inductor]

Description of the bug

Hi, I'm trying to use this plugin with rails and I get the following error but it looks very vague and not sure what's causing.

E, [2021-09-21T13:46:51.454756 #7] ERROR -- : OpenTelemetry error: unexpected configuration error due to uninitialized constant OpenTelemetry::Exporter - OpenTelemetry::SDK::ConfigurationError

My config is as follows

OpenTelemetry::SDK.configure do |c|
    c.use 'OpenTelemetry::Instrumentation::Rails'
    c.use 'OpenTelemetry::Instrumentation::Mysql2'
    c.use 'OpenTelemetry::Instrumentation::Redis'
    c.add_span_processor(
      OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
        exporter: OpenTelemetry::Exporter::Jaeger::AgentExporter.new(host: 'simplest-agent.observability', port: 6831)
      )
    )
    c.service_name = 'app'
    c.service_version = '0.1.0'
  end

I'm running this on EKS

Share details about your runtime

Operating system details: Linux, Ubuntu 20.04 LTS RUBY_ENGINE: "ruby" RUBY_VERSION: "2.7.2" RUBY_DESCRIPTION: "ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]"

Share a simplified reproduction if possible

[arielvalentin]

@inductor could you share how you are loading your dependencies and requiring the exporter?

E.g. Are you using a Gemfile + Bundler.require?

[inductor]

Hi @arielvalentin This is our changes https://github.com/cloudnativedaysjp/dreamkast/pull/859/files

I modified requires now, and got this error.

I think name resolve is working and so is my jaeger and now wondering what this error shows.

E, [2021-09-21T15:29:41.252817 #7] ERROR -- : OpenTelemetry error: unexpected configuration error due to exporter {:exporter=>#<OpenTelemetry::Exporter::Jaeger::AgentExporter:0x0000558abdd1ca70 @client=#<OpenTelemetry::Exporter::Jaeger::Thrift::Agent::Client:0x0000558abdd1c8e0 @iprot=#<Thrift::CompactProtocol:0x0000558abdd1c980 @trans=#<OpenTelemetry::Exporter::Jaeger::Transport:0x0000558abdd1ca48 @socket=#<UDPSocket:fd 6, AF_INET, 10.100.192.250, 41170>, @buffer=#<Thrift::MemoryBufferTransport:0x0000558abdd1c9f8 @buf="", @index=0>>, @last_field=[0], @boolean_value=nil, @rbuf="\x00\x00\x00\x00\x00\x00\x00\x00">, @oprot=#<Thrift::CompactProtocol:0x0000558abdd1c980 @trans=#<OpenTelemetry::Exporter::Jaeger::Transport:0x0000558abdd1ca48 @socket=#<UDPSocket:fd 6, AF_INET, 10.100.192.250, 41170>, @buffer=#<Thrift::MemoryBufferTransport:0x0000558abdd1c9f8 @buf="", @index=0>>, @last_field=[0], @boolean_value=nil, @rbuf="\x00\x00\x00\x00\x00\x00\x00\x00">, @seqid=0>, @max_packet_size=65000, @shutdown=false, @sizing_transport=#<OpenTelemetry::Exporter::Jaeger::AgentExporter::SizingTransport:0x0000558abdd1c890 @size=0>, @sizing_protocol=#<Thrift::CompactProtocol:0x0000558abdd1c868 @trans=#<OpenTelemetry::Exporter::Jaeger::AgentExporter::SizingTransport:0x0000558abdd1c890 @size=0>, @last_field=[0], @boolean_value=nil, @rbuf="\x00\x00\x00\x00\x00\x00\x00\x00">, @timeout=10.0>} does not appear to be a valid exporter - OpenTelemetry::SDK::ConfigurationError

[inductor]

netcat shows the port for this host is open

root@dreamkast-b8bcd97f8-xm9hc:/app# nc -v -u -z -w 3 simplest-agent.observability 6831
DNS fwd/rev mismatch: simplest-agent.observability.svc.cluster.local != 10-100-216-9.simplest-collector-headless.observability.svc.cluster.local

apiVersion: v1
kind: Service
metadata:
  labels:
    app: jaeger
    app.kubernetes.io/component: service-agent
    app.kubernetes.io/instance: simplest
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: simplest-agent
    app.kubernetes.io/part-of: jaeger
  name: simplest-agent
  namespace: observability
spec:
  clusterIP: None
  clusterIPs:
  - None
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: zk-compact-trft
    port: 5775
    protocol: UDP
  - name: config-rest
    port: 5778
  - name: jg-compact-trft
    port: 6831
    protocol: UDP
  - name: jg-binary-trft
    port: 6832
    protocol: UDP
  selector:
    app: jaeger
    app.kubernetes.io/component: all-in-one
    app.kubernetes.io/instance: simplest
    app.kubernetes.io/managed-by: jaeger-operator
    app.kubernetes.io/name: simplest
    app.kubernetes.io/part-of: jaeger

[arielvalentin]

Looking at the Gist you provided, I see you are setting the environment variable as well as manually configuring the exporter. I think you want to chose to configure this via ENV or configure it programmatically but not both.

https://github.com/cloudnativedaysjp/dreamkast/pull/859/files#diff-109bb4a1981c074f841c32f60bd7caa0ea2fd00fa720467cc76426a359aa6b52R5

https://github.com/cloudnativedaysjp/dreamkast/pull/859/files#diff-109bb4a1981c074f841c32f60bd7caa0ea2fd00fa720467cc76426a359aa6b52R16

Take a look here if you want to configure the Jaeger Collector or Agent:

https://github.com/open-telemetry/opentelemetry-ruby/tree/main/exporter/jaeger#how-do-i-get-started

[inductor]

I removed the ENV but error still is the same :|

https://github.com/cloudnativedaysjp/dreamkast/pull/859/files

E, [2021-09-21T16:51:33.000104 #7] ERROR -- : OpenTelemetry error: unexpected configuration error due to exporter {:exporter=>#<OpenTelemetry::Exporter::Jaeger::AgentExporter:0x000055701c1fa540 @client=#<OpenTelemetry::Exporter::Jaeger::Thrift::Agent::Client:0x000055701c1fa388 @iprot=#<Thrift::CompactProtocol:0x000055701c1fa450 @trans=#<OpenTelemetry::Exporter::Jaeger::Transport:0x000055701c1fa518 @socket=#<UDPSocket:fd 6, AF_INET, 10.100.187.247, 40161>, @buffer=#<Thrift::MemoryBufferTransport:0x000055701c1fa4a0 @buf="", @index=0>>, @last_field=[0], @boolean_value=nil, @rbuf="\x00\x00\x00\x00\x00\x00\x00\x00">, @oprot=#<Thrift::CompactProtocol:0x000055701c1fa450 @trans=#<OpenTelemetry::Exporter::Jaeger::Transport:0x000055701c1fa518 @socket=#<UDPSocket:fd 6, AF_INET, 10.100.187.247, 40161>, @buffer=#<Thrift::MemoryBufferTransport:0x000055701c1fa4a0 @buf="", @index=0>>, @last_field=[0], @boolean_value=nil, @rbuf="\x00\x00\x00\x00\x00\x00\x00\x00">, @seqid=0>, @max_packet_size=65000, @shutdown=false, @sizing_transport=#<OpenTelemetry::Exporter::Jaeger::AgentExporter::SizingTransport:0x000055701c1fa338 @size=0>, @sizing_protocol=#<Thrift::CompactProtocol:0x000055701c1fa2e8 @trans=#<OpenTelemetry::Exporter::Jaeger::AgentExporter::SizingTransport:0x000055701c1fa338 @size=0>, @last_field=[0], @boolean_value=nil, @rbuf="\x00\x00\x00\x00\x00\x00\x00\x00">, @timeout=10.0>} does not appear to be a valid exporter - OpenTelemetry::SDK::ConfigurationError

[arielvalentin]

@inductor I did not notice this before but it looks like you are using a keyword arg exporter however the BatchSpanExporter uses a positional argument. Remove the exporter keyword and try again:

In your snippet:

    c.add_span_processor(
      OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
        exporter: OpenTelemetry::Exporter::Jaeger::AgentExporter.new(host: 'simplest-agent.observability', port: 6831)
      )
    )

From the guide:

    c.add_span_processor(
      OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
         OpenTelemetry::Exporter::Jaeger::AgentExporter.new(host: 'simplest-agent.observability', port: 6831)
      )
    )

[arielvalentin]

Checking in with you @inductor. Did that fix your issue?

[inductor]

@arielvalentin Oh thank you, the trace data is there now!

One question, the service name is "unknown_service" while I set the service name by ENV['OTEL_SERVICE_NAME'] = 'dreamkast' and tracer = OpenTelemetry.tracer_provider.tracer('dreamkast', '0.1.0'). Are those not enough?

[inductor]

c.service_name = 'service_name' did this, but I didn't see it in the guide.

[arielvalentin]

The TracerProvider#tracer method created a "named tracer". That sets the instrumentation.name and instrumentation.version attributes and will not override the service.name and service.version.

Looking at your PR you are both setting the environment variable OTEL_SERVICE_NAME and the configurator.service_name. You only have to set one or the other, however that does not explain why it appears as unknown_service.

[arielvalentin]

I see now. You are creating a tracer inside of the configure block. That ends up creating a tracer that is not properly configured.

Create the tracer outside of the configure block.

[inductor]

@arielvalentin This approach works. thanks for your advice!

Btw I'm completely new to open telemetry and jeager, and I noticed that in the current setting I only see MySQL traces, but not app tracing in rails application while I have c.use 'OpenTelemetry::Instrumentation::Rails'. Do I have to make additional configurations for tracer? If there's a good reference that'd be great.

[arielvalentin]

@inductor My recommendation would be to start with enabling instrumentation-all and then disabling any instrumentation after that.

Please take a look at the quick start guide for more details: https://github.com/open-telemetry/opentelemetry-ruby/blob/main/website_docs/quick_start.md

Also I am going to convert this to a discussion since there is no bug to fix here.