OTEL Kubernetes Instrumentation env var OTEL_PYTHON_DISABLED_INSTRUMENTATIONS don't disable specified packages.

[AlejandroReBa]

Describe your environment

Platform environment Google Kubernetes Engine: 1.27.11-gke.1062000 OpenTelemetry Operator v.0.98.0 OpenTelemetry Contrib Collector - v0.98.0 OpenTelemetry Python Contributor - v0.44b0 OpenTelemetry Instrumentation GRPC - v0.44b0

List of OpenTelemetry packages installed:

opentelemetry-api==1.23.0
opentelemetry-distro==0.44b0
opentelemetry-exporter-otlp-proto-common==1.23.0
opentelemetry-exporter-otlp-proto-http==1.23.0
opentelemetry-instrumentation==0.44b0
opentelemetry-instrumentation-aio-pika==0.44b0
opentelemetry-instrumentation-aiohttp-client==0.44b0
opentelemetry-instrumentation-aiopg==0.44b0
opentelemetry-instrumentation-asgi==0.44b0
opentelemetry-instrumentation-asyncio==0.44b0
opentelemetry-instrumentation-asyncpg==0.44b0
opentelemetry-instrumentation-boto==0.44b0
opentelemetry-instrumentation-boto3sqs==0.44b0
opentelemetry-instrumentation-botocore==0.44b0
opentelemetry-instrumentation-cassandra==0.44b0
opentelemetry-instrumentation-celery==0.44b0
opentelemetry-instrumentation-confluent-kafka==0.44b0
opentelemetry-instrumentation-dbapi==0.44b0
opentelemetry-instrumentation-django==0.44b0
opentelemetry-instrumentation-elasticsearch==0.44b0
opentelemetry-instrumentation-falcon==0.44b0
opentelemetry-instrumentation-fastapi==0.44b0
opentelemetry-instrumentation-flask==0.44b0
opentelemetry-instrumentation-grpc==0.44b0
opentelemetry-instrumentation-httpx==0.44b0
opentelemetry-instrumentation-jinja2==0.44b0
opentelemetry-instrumentation-kafka-python==0.44b0
opentelemetry-instrumentation-logging==0.44b0
opentelemetry-instrumentation-mysql==0.44b0
opentelemetry-instrumentation-mysqlclient==0.44b0
opentelemetry-instrumentation-pika==0.44b0
opentelemetry-instrumentation-psycopg2==0.44b0
opentelemetry-instrumentation-pymemcache==0.44b0
opentelemetry-instrumentation-pymongo==0.44b0
opentelemetry-instrumentation-pymysql==0.44b0
opentelemetry-instrumentation-pyramid==0.44b0
opentelemetry-instrumentation-redis==0.44b0
opentelemetry-instrumentation-remoulade==0.44b0
opentelemetry-instrumentation-requests==0.44b0
opentelemetry-instrumentation-sklearn==0.44b0
opentelemetry-instrumentation-sqlalchemy==0.44b0
opentelemetry-instrumentation-sqlite3==0.44b0
opentelemetry-instrumentation-starlette==0.44b0
opentelemetry-instrumentation-system-metrics==0.44b0
opentelemetry-instrumentation-tornado==0.44b0
opentelemetry-instrumentation-tortoiseorm==0.44b0
opentelemetry-instrumentation-urllib==0.44b0
opentelemetry-instrumentation-urllib3==0.44b0
opentelemetry-instrumentation-wsgi==0.44b0
opentelemetry-propagator-aws-xray==1.0.1
opentelemetry-propagator-b3==1.23.0
opentelemetry-propagator-jaeger==1.23.0
opentelemetry-propagator-ot-trace==0.44b0
opentelemetry-proto==1.23.0
opentelemetry-sdk==1.23.0
opentelemetry-semantic-conventions==0.44b0
opentelemetry-test-utils==0.44b0
opentelemetry-util-http==0.44b0

Application Python 3.10.13 grpc-stubs==1.53.0.5 grpcio==1.62.2 grpcio-status==1.62.2

I followed steps defined on the OpenTelemetry doc for python autoinstrumentation via OpenTelemetry Operator Instrumentation Object

In this snippet I specify to disable opentelemetry-autoinstrumentation-grcp package. I also tried with other packages as django and celery and even when the environment variable is set in the python deployment, all packages appear as installed with pip list.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: otel-instrumentation
  namespace: default
spec:
  exporter:
    endpoint: http://agent-collector-collector.otel-collector:4317
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "1"

  python:
    env:
    # Python autoinstrumentation only supports OTLP/HTTP which the collector runs on port 4318,
    # see https://github.com/open-telemetry/opentelemetry-operator/issues/924
    - name: OTEL_PYTHON_DISABLED_INSTRUMENTATIONS
      value: grpc
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: http://agent-collector-collector.otel-collector:4318
    - name: OTEL_LOGS_EXPORTER
      value: otlp_proto_http
    - name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
      value: 'true'
    - name: OTEL_METRICS_EXPORTER
      value: none

I also tried to only disable the specific service

    - name: OTEL_PYTHON_GRPC_EXCLUDED_SERVICES
      value: 'my-deployment-instrumented-python-service'

but nothing changed, my application is still instrumented for grcp (just in case the issue may be related) I have been looking everywhere for days to detect if I am setting something wrong but I am blocked.

Steps to reproduce

1. Install OpenTelemetry Kubernetes Operator in the cluster
2. Deploy the OpenTelemetry Instrumentation Kubernetes Manifest shared above (replace the collector endpoint)
3. Deploy a python application deployment into Kubernetes with autoinstrumentation enabled instrumentation.opentelemetry.io/inject-python: "true" (the entry command can be a sleep infinity to avoid developing any python code, only python environment is required)

What is the expected behavior? When I check pip list opentelemetry-instrumentation-grcp package shouldn't be there (or any other package specified on OTEL_PYTHON_DISABLED_INSTRUMENTATIONS)

What is the actual behavior? opentelemetry-instrumentation-grcp package is still there.

Additional context My application fails when grpc instrumentation is enabled. If I run my python application manually inside the Kubernetes pod after pip uninstall opentelemetry-instrumentation-grcp everything work as expected.

I need to disable this from the Instrumentation Service, but neither OTEL_PYTHON_GRPC_EXCLUDED_SERVICES nor OTEL_PYTHON_DISABLED_INSTRUMENTATIONS seem to work properly with the shared configuration.

[AlejandroReBa]

Any feedback based on experience with OTEL_PYTHON_DISABLED_INSTRUMENTATIONS is very welcome, I still consider either it is a bug or I am missing any critical step.

[xrmx]

From a quick look it seems thatOTEL_PYTHON_DISABLED_INSTRUMENTATIONS is handled by opentelemetry-instrument that is not called using the auto instrumentation operator.

[AlejandroReBa]

Thanks for the reply @xrmx Do you think is there any possibility of enabling that flag when using the autoinstrumentation operator?

[xrmx]

Thanks for the reply @xrmx Do you think is there any possibility of enabling that flag when using the autoinstrumentation operator?

I took a look at the operator last week and the only thing is doing is setting environment variables. So it's not a substitute for opentelemetry-instrument. Since we have already problems with some specific frameworks (Django) with the operator because missing stuff we do in opentelemetry-instrument, to make more software work out of the box I think we would probably move some of the opentelemetry-instrument functionalities to other packages so that it's run even with the kubernetes operator.

[xrmx]

Taking a closer look at this I think I was wrong. OTEL_PYTHON_DISABLED_INSTRUMENTATIONS should be respected anyway because it will be handled by opentelemetry-instrumentation sitecustomize.py and not by the opentelemetry-instrument command.

BTW that environment variable means to disable an instrumentation not to not install or remove it.

[luong-komorebi]

I am seeing the same problem. Putting OTEL_PYTHON_DISABLED_INSTRUMENTATIONS inside the deployment or in Instrumentation CRD seems to have no effect when injecting autoinstrumentation

[xrmx]

Tested this with with a flask app with a requests call in a view and I am able to disable both the flask instrumentation and the requests one. The requests one is tricky because I have to also disable urllib3 one otherwise it'll be instrumented too.

For grpc the instrumentation entry point is called grpc_client and that may be your issue.

[luong-komorebi]

@xrmx this did ~not~ fix my issue. Specifically I set

OTEL_PYTHON_DISABLED_INSTRUMENTATIONS: "grpc_aio_server,grpc_server"

If this setting is correct, can you update the document to help users like us figure that out ourselves (without looking at the code)

In the example document, I can see

For example, if your Python program uses the redis and kafka-python packages, by default the agent will use the opentelemetry-instrumentation-redis and opentelemetry-instrumentation-kafka-python packages to instrument them. To disable this, you can set OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=redis,kafka-python.

I assume that it should be

OTEL_PYTHON_DISABLED_INSTRUMENTATIONS: "grpc"

there is no clue how to be exact about the disabled instrumentations' values

Thanks to your suggestion, I can self-figure it out. I guess that by looking at the code at [project.entry-points.opentelemetry_instrumentor], user will figure out themselves, but we should still have a clue so that users can self-serve

I will try to open a pull request but want to get your feedback if this should be done

[xrmx]

@luong-komorebi See https://github.com/open-telemetry/opentelemetry.io/pull/4866