OTLPSpanExporter does not export named trace spans to collector

[PaulBenn-UnlikelyAI]

Environment

# OS version (Windows CMD 'ver')
Microsoft Windows [Version 10.0.19042.1052]
# WSL version ('uname -r')
5.4.72-microsoft-standard-WSL2
# python --version
3.8.5
# pip --version
21.0.1
# pipenv --version
2020.11.15
# pipenv shell && pip freeze | grep opentelemetry
opentelemetry-api==1.3.0
opentelemetry-exporter-otlp==1.3.0
opentelemetry-exporter-otlp-proto-grpc==1.3.0
opentelemetry-proto==1.3.0
opentelemetry-sdk==1.3.0
opentelemetry-semantic-conventions==0.22b0

Steps to reproduce Set up a client:

1. Create new Python pipenv project.
2. Define Pipfile as follows:

   
   [requires]
   python_version = "3.8"

[packages] opentelemetry-api = "" opentelemetry-sdk = "" opentelemetry-exporter-otlp = "*"

3. Run `pipenv update` to install packages.
4. Define file `open_telemetry_bug.py`.
5. Populate file `open_telemetry_bug.py` with verbatim copy-paste of [latest documentation](https://opentelemetry-python.readthedocs.io/en/latest/exporter/otlp/otlp.html).

Set up a server:
1. Create `Dockerfile` elsewhere on filesystem as follows:
```dockerfile
# This bug is also reproducible with aws-otel-collector. See:
# https://github.com/aws-observability/aws-otel-collector/blob/main/docs/developers/docker-demo.md
FROM otel/opentelemetry-collector-contrib:0.29.0
COPY otel-agent-config.yaml /etc/otel/config.yaml

2. Create otel-agent-config.yaml in the same directory as follows (zpages is not necessary, but aids the example):

   ---
   extensions:
   zpages:
   endpoint: 0.0.0.0:55679
   receivers:
   otlp:
   protocols:
     grpc:
       endpoint: 0.0.0.0:4317
     http:
       endpoint: 0.0.0.0:55681
   exporters:
   logging:
   loglevel: debug
   service:
   pipelines:
   metrics:
     receivers: [otlp]
     processors: []
     exporters: [logging]
   extensions: [zpages]

3. Build and run container. Remove options as you see fit.

   docker build --rm --no-cache -t otel-collector . && docker run -e AWS_REGION=eu-west-2 -p 4317:4317 -p 55679:55679 -p 55680:55680 -p 55681:55681 -p 8888:8888 otel-collector --config=/etc/otel/config.yaml --log-level DEBUG

Send a metric:

1. Go to your client's directory and execute:

   pipenv shell
   python -m open_telemetry_bug

What is the expected behavior? Given that this is the official example on the documentation, I expect it to work without any additional configuration:

1. Valid span with the name foo is created and instantly ended.
2. Span foo is emitted from client OTLPSpanExporter.
3. Span foo reaches collector.
4. Collector logging exporter prints it out.
5. zpages reflects the span.
6. Prometheus endpoint on :8888 reflects the span.

What is the actual behavior?

1. Valid span with the name foo is created and instantly ended.
2. Span foo is either not emitted correctly from client OTLPSpanExporter, or not emitted at all. Adding a SimpleSpanProcessor(ConsoleSpanExporter()) prints it.
3. Span foo never reaches collector. Only a span of name opentelemetry.proto.collector.trace.v1.TraceService.Export reaches collector.
4. Collector logging exporter never prints the above (intermediate?) span out.
5. zpages reflects the above span as Running. This state never changes.
6. Prometheus endpoint on :8888 does not reflect the span.

Additional context This is a verbatim example from various pieces of documentation, wrapped in a thin pipenv layer. Am I doing something wrong? If not, it would be good if we could update the documentation to reflect an example which works out of the box

[lzchen]

Try adding [batch] to processors in your config?

[PaulBenn-UnlikelyAI]

Hi @Izchen, I've tried this configuration with ([]) and without ([batch]), but the result is the same.

[lzchen]

@PaulBenn-UnlikelyAI You also added a

processors:
  batch:

section?

[PaulBenn-UnlikelyAI]

@Izchen yes, that's correct. I've also played around with multiple variations of the batch processor's settings, including limiting the batch size to 1.

[lzchen]

I ran the example (verbatim) in my local environment and the span is printed successfully.

I have the following in my otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  logging:
    loglevel: debug

processors:
  batch:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]

as well as docker-compose-yml

version: "2"
services:

  otel-collector:
    image: otel/opentelemetry-collector-dev:latest
    command: ["--config=/etc/otel-collector-config.yaml", "${OTELCOL_ARGS}"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"       # OTLP gRPC receiver

[lzchen]

Simply running docker-compose up works with this example.

[PaulBenn-UnlikelyAI]

Hi @Izchen, this is using a different image. I am using a Dockerfile with:

FROM otel/opentelemetry-collector-contrib:0.29.0

...whereas you are using a docker-compose file with:

image: otel/opentelemetry-collector-dev:latest

Could you please clarify the relationship between these images, and their difference to the AWS version, if applicable? I cannot find any relevant comparison documentation, and it would be good to have an official recommendation from you!

[lzchen]

@PaulBenn-UnlikelyAI

Our getting started docs point to using the latest version of opentelemetry-collector (not contrib).

[PaulBenn-UnlikelyAI]

Thanks - I had a brief look just now, and I'll try this tomorrow, but could you please clarify the relationship between opentelemetry-collector-contrib, opentelemetry-collector-dev and opentelemetry-collector?

[lzchen]

I would ask this in the OpenTelemetry collector repository.

[PaulBenn-UnlikelyAI]

Hi @Izchen. I have confirmed that your example above works perfectly with opentelemetry-collector-dev:latest and the span is printed by the logging exporter, as expected. I am still quite confused as to the relationship between these three images, so I have opened

https://github.com/open-telemetry/opentelemetry-collector/issues/3669#issue-947099130

In the OTel Collector repository. Thank you for your help, and feel free to resolve this issue.