Open ilyamochalov opened 10 months ago
Unlike dotnet, I believe this is a fault of the docker image we supply, not the instrumentation itself.
@open-telemetry/operator-approvers I think we need to make a concrete decision on what auto-instrumentation images we supply. For all appropriate languages, will will supply both musl and glibc based images? Or is dotnet a one-off case because of how the dotnet agent is supplied?
@TylerHelmuth thank you for checking this issue.
psutil_linux.abi3.so: __sched_cpufree: symbol not found
and similar error messages indicate that psutil package (which is a dependency of Python OTel packages) was installed against a system with different C lib implementation (Glibc vs Musl). When pip installing psutil
CPython compiles something against C lib. Pip dependencies compiled against Glibc won't work on Musl systems
Final autoinstrumentation images for .NET, Python, and other languages are simply one way to distribute programming language-specific auto-instr libraries. I think for languages which runtime depend on system C Lib we need to build auto-instr libraries against both Glibc and Musl libraries and bring both sets of artifacts to application. Then OTel Kubernetes operator should make a decision about what artifact needs to be injected into the app container.
We discussed this issue during the SIG call today. We'd like to have a clean solution that auto-detects which libs to use and handles everything for the user, but we think finding a solution like that is unlikely.
Most likely we have to implement a dotnet-like solution where the user can specify the libs they need.
@srikanthccv do you or any other Python maintainers have any advice on how to handle this?
I took a brief look at the dotnet solution. I think the same should work for Python as well. I will take some time to review the instrumentation side and see if there are any cases that require special handling.
@srikanthccv thank you for taking a look. I will proceed with my PR proposing changes to operator and instr docker image (please review dockerfile on the PR link above)
@open-telemetry/operator-approvers PR is ready, can someone review it please https://github.com/open-telemetry/opentelemetry-operator/pull/2266?
Bumped into the psutil stacktrace issue while exploring python autoinstrumentation as defined by the files in the e2e-instrumentation/instrumentation-python directory.
Looks like the dockerfile for the default init container and for the test app (published at ghcr.io/open-telemetry/opentelemetry-operator/e2e-test-app-python:main
) use binary incompatible base images -- one uses python3.11 (glibc) and the other alpine.318 (musl).
Also, the collector configs defined in the instrumentation directories (e.g. tests/e2e-instrumentation/instrumentation-python/00-install-collector.yaml) don't specify a metrics receiver, but python auto-instrumentation sends metrics, so you get a 404 in the logs because of the failed metrics exports. Adding a metrics receiver to the collector pipeline solves the problem.
Component(s)
instrumentation
Is your feature request related to a problem? Please describe.
Python autoinstrumentation for musl libc based application containers fails with the following error:
Root cause: current autoinstrumentation build packaged for BSD libc.
Describe the solution you'd like
Describe alternatives you've considered
No response
Additional context
Similar change was made for .Net