Auto instrumentation is not working

[gyliu513]

Following the steps here https://opentelemetry.io/docs/instrumentation/python/automatic/example/#execute-the-automatically-instrumented-server

Run server first

opentelemetry-instrument --traces_exporter console --metrics_exporter none python server_automatic.py

Then run client

python client.py testing
{
    "name": "client-server",
    "context": {
        "trace_id": "0x8212674bb3c491c6ca3995ba997d90bc",
        "span_id": "0xe8254224de6581ba",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": "0x71f65d5cce1f1f63",
    "start_time": "2023-10-25T17:24:31.980542Z",
    "end_time": "2023-10-25T17:24:31.992175Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.20.0",
            "service.name": "unknown_service"
        },
        "schema_url": ""
    }
}
{
    "name": "client",
    "context": {
        "trace_id": "0x8212674bb3c491c6ca3995ba997d90bc",
        "span_id": "0x71f65d5cce1f1f63",
        "trace_state": "[]"
    },
    "kind": "SpanKind.INTERNAL",
    "parent_id": null,
    "start_time": "2023-10-25T17:24:31.980484Z",
    "end_time": "2023-10-25T17:24:31.992201Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {},
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.20.0",
            "service.name": "unknown_service"
        },
        "schema_url": ""
    }
}

Client do have output, but server automatic do not have any output, based on the doc here https://opentelemetry.io/docs/instrumentation/python/automatic/example/#execute-the-automatically-instrumented-server

The console running server_automatic.py will display the spans generated by instrumentation as JSON. The spans should appear similar to the following example:

[kquinsland]

I have hit the same issue while trying to debug auto-instrumentation with an internal non-flask app.

Even when explicitly setting arguments via the CLI, I get nothing in either the console or the debug logs of the otel-collector that I have running locally:

❯ python3 --version
Python 3.12.0
❯ opentelemetry-instrument \
    --traces_exporter console,otlp \
    --metrics_exporter console,otlp \
    --service_name your-service-name \
    --exporter_otlp_endpoint 127.0.0.1:4317 \
    --exporter_otlp_insecure=true \
    python3 server_automatic.py
 * Tip: There are .env or .flaskenv files present. Do "pip install python-dotenv" to use them.
 * Serving Flask app 'server_automatic'
 * Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:8082
Press CTRL+C to quit
testing
127.0.0.1 - - [13/Nov/2023 12:43:45] "GET /server_request?param=testing HTTP/1.1" 200 -

I do see json dumped to the console when using the client.py though.

---

I was wondering if the 3.12 was breaking things and changed venv to 3.11.6 and still nothing:

❯ python3 --version
Python 3.11.6
❯ opentelemetry-instrument --version
opentelemetry-instrument 0.42b0
❯ opentelemetry-instrument \
    --traces_exporter console,otlp \
    --metrics_exporter console,otlp \
    --service_name your-service-name \
    --exporter_otlp_endpoint 127.0.0.1:4317 \
    --exporter_otlp_insecure=true \
    python3 server_automatic.py
 * Tip: There are .env or .flaskenv files present. Do "pip install python-dotenv" to use them.
 * Serving Flask app 'server_automatic'
 * Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:8082
Press CTRL+C to quit
testing
127.0.0.1 - - [13/Nov/2023 13:00:34] "GET /server_request?param=testing HTTP/1.1" 200 -

[pcwiese]

+1. The other 2 samples work just fine, but I get nothing on the server console when hitting server_request using automatic instrumentation.

[h1t35h]

+1 Seems something have changed auto instrumentation isn't working for me as well. Even FastAPI auto instrumentation is broken.

[keisukesakasai]

I have hit the same issue with opentelemetry-instrumentation 0.42b0.

Will it work using 0.41b0?

[panys9]

I can only make it work with python 3.7

[flands]

The issue appears to be that Flask/Werkzeug 3.0+ is not supported by the auto instrumentation. If you install an older version, then it works as expected: pip uninstall Flask Werkzeug; pip install "Flask <3" "Werkzeug <3"

[flands]

Directions can be updated to reflect current state until https://github.com/open-telemetry/opentelemetry-python-contrib/issues/1975 is resolved.

[gsakkis]

It's not just Flask, I can't get any ASGI framework auto instrumentation to work; I tried FastAPI, Starlette, plain ASGI.

FastAPI example:

# app.py

from fastapi import FastAPI

app = FastAPI()

@app.get("/")
def read_root():
    return {"Hello": "World"}

Run server: opentelemetry-instrument --traces_exporter console --metrics_exporter none uvicorn app:app Run client: curl 'localhost:8000/' Installed packages:

$ pip freeze | grep opentelemetry-instrumentation

opentelemetry-instrumentation==0.43b0
opentelemetry-instrumentation-asgi==0.43b0
opentelemetry-instrumentation-fastapi==0.43b0
opentelemetry-instrumentation-flask==0.43b0
opentelemetry-instrumentation-httpx==0.43b0
opentelemetry-instrumentation-starlette==0.43b0
opentelemetry-instrumentation-wsgi==0.43b0

[aeb-dev]

Is there any update on this. We are having the same issue as @gsakkis

[gsakkis]

@aeb-dev apparently auto instrumentation needs the opentelemetry-distro package as well, I opened a separate issue about it.

[aeb-dev]

I actually have opentelemetry-distro and it still does not work

[fromagge]

I was facing this even with the minimal example and reading the documentation is realized that I had missed this:

https://opentelemetry.io/docs/languages/python/automatic/example/#instrumentation-while-debugging

Basically on Flask when using debug=True you have to use use_reloader=False, if not the instrumentation may not worked.

This worked for me, I overlooked it.

[ijkbytes]

I actually have opentelemetry-distro and it still does not work

@aeb-dev Did you finally find the cause of this issue? I'm also troubled in this.

[emdneto]

@gyliu513 Gentle remind. Any reason to keep the issue open?

[aeb-kb01]

@ijkbytes It happens when you use an asgi server with multiple processes

[DrLuke]

If you use uvicorn with reloading like uvicorn.run("myapp:app", host="localhost", port=8000, reload=True) it will also not work. Set reload to False.