Python - Allow auto installation in k8s clusters

estolfo commented 3 months ago

Some work should go upstream, here are the relevant issues:

xrmx commented 2 months ago

PR for cleanup the k8s operator handling of used OTLP protocol https://github.com/open-telemetry/opentelemetry-operator/pull/3165

xrmx commented 1 month ago

Wondering if running django as python -m django runserver instead of python manage.py runserver would make a difference as per https://docs.python.org/3/library/sys.html#sys.path we should get the current working directory in sys.path if we call our script that way. The question is that sys.path is relevant for django.

xrmx commented 3 weeks ago

Built a custom docker image for the k8s operator and it's evident that for Python this won't be a silver bullet. Problem is that a lot of libraries depends on C extensions that are platform dependant.

For example psutil (from system metrics instrumentation) built on an alpine distro but run into an ubuntu container will fail because of:

ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

and this import (File "/otel-auto-instrumentation-python/elasticotel/distro/__init__.py", line 25, in <module>\nfrom opentelemetry.instrumentation.system_metrics import () comes from our distro so it should be fixed.

grpc aswell:

ImportError: cannot import name 'cygrpc' from 'grpc._cython' (/otel-auto-instrumentation-python/grpc/_cython/__init__.py)

From a different path though:

  File "/otel-auto-instrumentation-python/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 39, in initialize
    _load_instrumentors(distro)
  File "/otel-auto-instrumentation-python/opentelemetry/instrumentation/auto_instrumentation/_load.py", line 91, in _load_instrumentors
    raise exc
  File "/otel-auto-instrumentation-python/opentelemetry/instrumentation/auto_instrumentation/_load.py", line 87, in _load_instrumentors
    distro.load_instrumentor(entry_point, skip_dep_check=True)

At the moment a failure of an instrumentor makes the whole process fail and in this scenario (or even generally?) it's not want we want.

xrmx commented 3 weeks ago

So at the moment we are building one image based on the Python 3.12 chainguard image (glibc based). In order to cover more environments we should probably increase the matrix of the images we build with more python versions and/or with musl based base images.

xrmx commented 1 week ago

A few otel operator related PRs to improve things:

[x] one to set a default for logs exporting if one decide to enable them https://github.com/open-telemetry/opentelemetry-operator/pull/3330
[ ] one to handle musl based containers, if accepted we can work out of the box https://github.com/open-telemetry/opentelemetry-operator/pull/3332
[x] the docs one documenting the issue with binary wheels https://github.com/open-telemetry/opentelemetry.io/pull/5338

xrmx commented 4 days ago

Closing this and using https://github.com/elastic/elastic-otel-python/issues/158 for the first round of followup improvements

elastic / elastic-otel-python

Python - Allow auto installation in k8s clusters #82