Python auto instrumentation not pushing traces to opentelemetry collector

[pankdhnd]

Describe your environment

I have setup an opetelemetry collector pod, and a jaeger pod (allinone). I am able to send tracer to collector using some demo apps (java based), but no traces are being sent via python instrumentation. I tried both, auto and manual instrumentation but nothing works out.

I am setting env variables as below

export OTEL_TRACES_EXPORTER=otlp
export OTEL_RESOURCE_ATTRIBUTES="service.name=demo-flask"
export OTEL_SERVICE_NAME="demo-flask"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4317"

On the above endpoint, I am able to send traces via java based demo application, but doesn't work with python. I also tried configuring http and zipkin, it does not work. Only thing that works is exporting the traces to console

Here is the python app I am using

from flask import Flask
from opentelemetry.instrumentation.flask import FlaskInstrumentor

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)

@app.route('/request')
def index():
    print("***Welcome***")
    return 'This is a sample Web App with Python Flask!'

if __name__ == "__main__":
    app.run(debug=False, port=5000)

here is the requirement.txt

backoff==1.10.0
certifi==2021.10.8
charset-normalizer==2.0.10
click==8.0.3
Deprecated==1.2.13
Flask==2.0.2
googleapis-common-protos==1.54.0
grpcio==1.43.0
idna==3.3
itsdangerous==2.0.1
Flask-Login==0.4.0
Flask-Migrate==2.0.2
Flask-Script==2.0.5
Flask-SQLAlchemy==2.4.0
Flask-WTF==0.14.2
Flask-User==1.0.1.5
pytest==3.0.5
pytest-cov==2.4.0
Jinja2==3.0.3
MarkupSafe==2.0.1
opentelemetry-api==1.12.0rc1
opentelemetry-exporter-otlp==1.12.0rc1
opentelemetry-exporter-otlp-proto-grpc==1.12.0rc1
opentelemetry-exporter-otlp-proto-http==1.12.0rc1
opentelemetry-exporter-zipkin
opentelemetry-exporter-zipkin-json
opentelemetry-exporter-zipkin-proto-http
opentelemetry-exporter-jaeger
opentelemetry-distro==0.31b0
opentelemetry-instrumentation==0.31b0
opentelemetry-instrumentation-dbapi==0.31b0
opentelemetry-instrumentation-flask==0.31b0
opentelemetry-instrumentation-psycopg2==0.31b0
opentelemetry-instrumentation-requests==0.31b0
opentelemetry-instrumentation-wsgi==0.31b0
opentelemetry-proto==1.12.0rc1
opentelemetry-sdk==1.12.0rc1
opentelemetry-semantic-conventions
opentelemetry-util-http==0.31b0
protobuf==3.19.2
requests==2.27.1
six==1.16.0
urllib3
Werkzeug==2.0.2
wrapt==1.13.3

I have setup opentelemetry collector pod with below configuration:

receivers:
  zipkin:
    endpoint: 0.0.0.0:9411
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch:
  memory_limiter:
    # 80% of maximum memory up to 2G
    limit_mib: 1500
    # 25% of limit up to 2G
    spike_limit_mib: 512
    check_interval: 5s
extensions:
  zpages: {}
  memory_ballast:
    # Memory Ballast size should be max 1/3 to 1/2 of memory.
    size_mib: 683
exporters:
  logging:
    loglevel: debug
  jaeger:
    endpoint: jaeger-collector-headless:14250
    tls:
      insecure: true
      insecure_skip_verify: true
service:
  extensions: [zpages, memory_ballast]
  pipelines:
    traces:
      receivers: [zipkin,otlp]
      processors: [batch,memory_limiter]
      exporters: [logging,jaeger]

Here is the dockerfile I am using to generate the image:

FROM python:latest
COPY requirements.txt .
RUN pip3 install -r requirements.txt
COPY test.py .
RUN opentelemetry-bootstrap --action=install
EXPOSE 5000
CMD [ "opentelemetry-instrument", "flask", "run", "--host=0.0.0.0" ]

Command that I am using to run the application opentelemetry-instrument falsk run

Steps to reproduce

1. Create opentelemetry collector pod, jaeger pod.
2. Set the env variables
3. Run the python app and hit it from browser/jmeter

What is the expected behavior? Traces are sent to opentelemetry collector, which forwards it to jaeger backend

What is the actual behavior? No traces are sent. I see no logs in opentelemetry collector pod

Additional context I have already configured opentelemetry-distro. I have also run opentelemetry-bootstrap --action=install before running the app. I tried different version of sdk (1.8, 1.11.1 and 1.12rc), but I see same behaviour.

I would really appreciate some help here.

[ocelotl]

Can you try this example and let us know how it goes?

[pankdhnd]

@ocelotl I tried the configuration from the example you gave, but unfortunately it did not work. Again, the other demo app (java) I have running in another pod can successfully trigger traces to collector, but not the pod with python app.

I set the opentelemetry environment variables from the python app kubernetes deployment.

[srikanthccv]

If you are running with k8s operator auto instrumentation you should use HTTP exporter for this stated reason.

export OTEL_TRACES_EXPORTER=otlp_proto_http
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

[pankdhnd]

@srikanthccv I just tried with k8s operator with instrumentation resource, but that didn't work as well. Below is the CR for instrumentation

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://my-otelcol-collector:4318

Here is the deployment I have configured for my application, where I am passing the env variables:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-python: "true"
  labels:
    app: test
    demo: "true"
  name: test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - image: flaskdemo:0.0.4
        imagePullPolicy: Always
        name: test
        env:
        - name: OTEL_TRACES_EXPORTER
          value: otlp_proto_http
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "service.name=demo-flask"
        - name: OTEL_SERVICE_NAME
          value: "demo-flask"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: http://my-otelcol-collector:4318
        ports:
        - containerPort: 5000
        resources: {}
      restartPolicy: Always

[pankdhnd]

Guys is there any solution available for this issue? If trace export is not working on Python, then I guess I will have to try some other programming language for my demo application

[sanketmehta28]

Can you add below env variables as well: OTEL_EXPORTER_OTLP_PROTOCOL=grpc OTEL_EXPORTER_OTLP_INSECURE=true

I have a fastapi app and able to send the traces to jaeger/zipkin via collector. Let me know if this helps

[pankdhnd]

@sanketmehta28, I tried supplying both the env variables, but still I am not able to see any activity in the collector. I wrote another app in golang, and got the traces in very first attempt.

[nreddipalle]

I am facing the same issue.. using fastapi auto instrumentation in k8S FastAPIInstrumentor.instrument_app(app)

with these environment variables

       - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: >-
            http://opentelemetry-collector.opentelemetry-collector.svc.cluster.local:4317
        - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
          value: >-
            http://opentelemetry-collector.opentelemetry-collector.svc.cluster.local:4317
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: grpc
        - name: OTEL_EXPORTER_OTLP_INSECURE
          value: true
        - name: OTEL_LOG_LEVEL
          value: debug
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: service.name=ness-platform

with no luck..

[srikanthccv]

The OTEL_EXPORTER_OTLP_PROTOCOL is not supported because we have separate packages for the each protocol.

 FastAPIInstrumentor.instrument_app(app)

This alone doesn't produce the telemetry. Are you using auto-instrumentation or did you setup the SDK pipeline manually? How are you running the fastapi?

[nreddipalle]

@srikanthccv plan is to use auto-instrumentation following this https://opentelemetry-python-contrib.readthedocs.io/en/latest/instrumentation/fastapi/fastapi.html

so I just added the below packages to my project

opentelemetry-distro = "^0.29b0"
opentelemetry-instrumentation-fastapi = "^0.29b0"
opentelemetry-exporter-otlp = "^1.10.0"

and poetry package manager pulls all the needed dependent telemetry packages

and added FastAPIInstrumentor.instrument_app(app) within main.py file.

from fastapi import Depends, FastAPI, Request
from fastapi.openapi.docs import get_redoc_html
from fastapi.openapi.utils import get_openapi
from fastapi.responses import JSONResponse
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

app = FastAPI(
    title="",
    version="1.0.0",
    docs_url=None,
    redoc_url=None,
    openapi_url=None,
)

 FastAPIInstrumentor.instrument_app(app)

and added the above environment variables as stated..

plan is to generate traces and send to opentelemetry collector runnning in k8S

Followed this example: https://github.com/SigNoz/sample-fastAPI-app

[srikanthccv]

Did you use the opentelemetry-instrument command?

[nreddipalle]

I did not.. let me try and update. Are these the steps to follow: pip install opentelemetry-launcher opentelemetry-instrument

[srikanthccv]

What is opentelemetry-launcher? We don't have any such package from official OpenTelemetry. Here is the getting started guide that will help you understand overall concepts https://opentelemetry.io/docs/instrumentation/python/getting-started/. And here is the automatic instrumentation that shows how to emit traces to console https://opentelemetry.io/docs/instrumentation/python/automatic/. You would replace the console exporter with OTLP exproter.

[ocelotl]

What is opentelemetry-launcher? We don't have any such package from official OpenTelemetry. Here is the getting started guide that will help you understand overall concepts https://opentelemetry.io/docs/instrumentation/python/getting-started/. And here is the automatic instrumentation that shows how to emit traces to console https://opentelemetry.io/docs/instrumentation/python/automatic/. You would replace the console exporter with OTLP exproter.

Just for the record opentelemetry-launcher is a Lightstep distro.

[srikanthccv]

@pankdhnd is this still an issue?

[j-flat]

i have the same issue, no traces send from the python app (using Tornado framework with opentelemetry-instrumentation-tornado installed). Trying to use the auto-instrumentation mode. I also had no troubles using sample Java-app, but with Python no luck.

[j-flat]

Update on my case, I managed to get the setup running but I found out that I cannot run the python backend apps on refresh/hot-reload mode, or the traces would not be captured. I could not find it quickly in documentation that this would not be supported, but in my case when I ran the backend containers in "production" mode, traces were exported normally.

[pankdhnd]

@srikanthccv, the issue still exists.

[TylerHelmuth]

@pankdhnd is the issue still happening on 1.14.0 and opentelemetry-contrib v0.35b0?

[andreacimino]

I have the same issue with these packages, tried everything.

opentelemetry-api==1.15.0 opentelemetry-instrumentation==0.36b0 opentelemetry-instrumentation-aws-lambda==0.36b0 opentelemetry-instrumentation-dbapi==0.36b0 opentelemetry-instrumentation-flask==0.36b0 opentelemetry-instrumentation-jinja2==0.36b0 opentelemetry-instrumentation-logging==0.36b0 opentelemetry-instrumentation-requests==0.36b0 opentelemetry-instrumentation-sqlalchemy==0.36b0 opentelemetry-instrumentation-sqlite3==0.36b0 opentelemetry-instrumentation-system-metrics==0.36b0 opentelemetry-instrumentation-tortoiseorm==0.36b0 opentelemetry-instrumentation-urllib==0.36b0 opentelemetry-instrumentation-urllib3==0.36b0 opentelemetry-instrumentation-wsgi==0.36b0 opentelemetry-propagator-aws-xray==1.0.1 opentelemetry-sdk==1.15.0 opentelemetry-semantic-conventions==0.36b0 opentelemetry-util-http==0.36b0

[andreacimino]

When running this flask app with the ' opentelemetry-instrument flask run' command, the logs are being collected (FlaskInstrumentor disabled). When enabling the flask instrumentor by uncommenting the proper line (and running only 'flask run'), nothing is collected. Tested with python 3.10.

from random import randint
from flask import Flask, request

from opentelemetry.instrumentation.flask import FlaskInstrumentor

app = Flask(__name__)

#FlaskInstrumentor().instrument_app(app)

@app.route("/rolldice")
def roll_dice():
    return str(do_roll())

def do_roll():
    return randint(1, 6)  # # These are the necessary import declarations

[srikanthccv]

@andreacimino

Note

You need to install a distro package to get auto instrumentation working. The opentelemetry-distro package contains the default distro and automatically configures some of the common options for users. For more info about opentelemetry-distro check here

https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/opentelemetry-instrumentation#installation

[andreacimino]

Tried again with the following dependecies (note that the distro package is installed), still not working:

backoff==2.2.1 certifi==2022.12.7 charset-normalizer==2.1.1 click==8.1.3 Deprecated==1.2.13 Flask==2.2.2 googleapis-common-protos==1.57.0 grpcio==1.51.1 idna==3.4 itsdangerous==2.1.2 Jinja2==3.1.2 MarkupSafe==2.1.1 opentelemetry-api==1.15.0 opentelemetry-distro==0.36b0 opentelemetry-exporter-otlp==1.15.0 opentelemetry-exporter-otlp-proto-grpc==1.15.0 opentelemetry-exporter-otlp-proto-http==1.15.0 opentelemetry-instrumentation==0.36b0 opentelemetry-instrumentation-aws-lambda==0.36b0 opentelemetry-instrumentation-dbapi==0.36b0 opentelemetry-instrumentation-flask==0.36b0 opentelemetry-instrumentation-grpc==0.36b0 opentelemetry-instrumentation-jinja2==0.36b0 opentelemetry-instrumentation-logging==0.36b0 opentelemetry-instrumentation-requests==0.36b0 opentelemetry-instrumentation-sqlite3==0.36b0 opentelemetry-instrumentation-urllib==0.36b0 opentelemetry-instrumentation-urllib3==0.36b0 opentelemetry-instrumentation-wsgi==0.36b0 opentelemetry-propagator-aws-xray==1.0.1 opentelemetry-proto==1.15.0 opentelemetry-sdk==1.15.0 opentelemetry-semantic-conventions==0.36b0 opentelemetry-util-http==0.36b0 protobuf==4.21.11 requests==2.28.1 typing_extensions==4.4.0 urllib3==1.26.13 Werkzeug==2.2.2 wrapt==1.14.1

[andreacimino]

@srikanthccv

Please note that with the 'opentelemetry-instrument flask run ' everything working, but when using the flask middleware, traces are not collected.

[TylerHelmuth]

@andreacimino to clarify, are the traces not correctly received by the collector or are they not created at all? If you use the console exporter what happens?

[andreacimino]

@TylerHelmuth

Finally it seems i solved the issue, but I think that something should be better documented, or maybe I am missing something. In case

from opentelemetry.instrumentation.flask import FlaskInstrumentor

  app = Flask(__name__)

  FlaskInstrumentor().instrument_app(app)

is enabled, the exporter configuration must be properly enabled anyway in order to make everything work.

 FlaskInstrumentor().instrument_app(app)

  from opentelemetry import trace
  from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
  from opentelemetry.sdk.resources import SERVICE_NAME, Resource
  from opentelemetry.sdk.trace import TracerProvider
  from opentelemetry.sdk.trace.export import BatchSpanProcessor

  resource = Resource(attributes={
    SERVICE_NAME: "your-service-name"
  })

  provider = TracerProvider(resource=resource)
  processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://127.0.0.1:4317"))
  provider.add_span_processor(processor)
  trace.set_tracer_provider(provider)

In case this last part is omitted, nothing is exported. There are three cases basically

1) Run opentelemetry-instrument flask run command with no configuration: everything works 2) Run flask run command with no configuration: FlaskInstrumentor().instrument_app(app) line: nothing is working 3) Run flask run command with explicit exported configuration : FlaskInstrumentor().instrument_app(app) line: everything is collected

I made a minimal example showing the behaviour on my machine.

test.zip

Run the collector with this command:

docker run -p 4317:4317 -v /PATH_TO_COLLECTOR/otel-collector-config.yaml:/etc/otel-collector-config.yaml otel/opentelemetry-collector:latest --config=/etc/otel-collector-config.yaml

I hope everything is little more clear now.

Thank you.

[lzchen]

Closing this issue as it seems to be resolved.