OpenTelemetry Python Auto-Instrumentation Image (v0.47b) Not Compatible with Seldon Core (v1.17.1) Image

ssurbhi19 commented 5 days ago

Describe your environment

OS: ios Python version: Python 3.11 Opentelemetry Python auto-instrumentation image: 0.47b Seldon Core: v1.17.1

What happened?

Previously, the OpenTelemetry Python auto-instrumentation image v0.44b was compatible with the Seldon Core base image seldonio/seldon-core-s2i-python3:1.15.0. However, due to security vulnerabilities, we upgraded the Python Auto-Instrumentation Image to the latest version, 0.47b. After the upgrade, we are encountering compatibility issues, and the application with the base image seldonio/seldon-core-s2i-python3:1.17.0 is not starting at all.

The error that we are encountering: `+ '[' -e /security-filter/fico_security_util-3-py3-none-any.whl ']'

echo 'INSTALLING SECURITY FILTER' INSTALLING SECURITY FILTER
pip install /security-filter/fico_security_util-3-py3-none-any.whl Processing /security-filter/fico_security_util-3-py3-none-any.whl Installing collected packages: fico-security-util Attempting uninstall: fico-security-util Found existing installation: fico_security_util 1.0 Uninstalling fico_security_util-1.0: Successfully uninstalled fico_security_util-1.0 Successfully installed fico-security-util-3

[notice] A new release of pip is available: 24.0 -> 24.2 [notice] To update, run: pip install --upgrade pip

'[' -z x ']'
'[' '!' false = true ']'
echo APPD_ENABLE set to false, skipping pyagent
/s2i/bin/run APPD_ENABLE set to false, skipping pyagent starting microservice 2024-08-28 06:46:09,353 INFO [root] [security_util.py:41] [trace_id=0 span_id=0 resource.service.name=mlx-app-design-py311-iris-9847-uxbwnvj2-2prlf9glng trace_sampled=False] - ########### log effective level was: INFO 2024-08-28 06:46:09,354 DEBUG [root] [security_util.py:43] [trace_id=0 span_id=0 resource.service.name=mlx-app-design-py311-iris-9847-uxbwnvj2-2prlf9glng trace_sampled=False] - ########### log effective level is now: DEBUG 2024-08-28 06:46:09,354 DEBUG [root] [security_util.py:47] [trace_id=0 span_id=0 resource.service.name=mlx-app-design-py311-iris-9847-uxbwnvj2-2prlf9glng trace_sampled=False] - ########### Security version: 3 ########### 2024-08-28 06:46:09,354 DEBUG [root] [security_util.py:63] [trace_id=0 span_id=0 resource.service.name=mlx-app-design-py311-iris-9847-uxbwnvj2-2prlf9glng trace_sampled=False] - Cache Size: 500 2024-08-28 06:46:09,354 DEBUG [root] [security_util.py:63] [trace_id=0 span_id=0 resource.service.name=mlx-app-design-py311-iris-9847-uxbwnvj2-2prlf9glng trace_sampled=False] - Cache Size: 500 2024-08-28 06:46:09,354 DEBUG [root] [token_interceptor.py:38] [trace_id=0 span_id=0 resource.service.name=mlx-app-design-py311-iris-9847-uxbwnvj2-2prlf9glng trace_sampled=False] - ########### log effective level was: DEBUG 2024-08-28 06:46:09,354 DEBUG [root] [token_interceptor.py:40] [trace_id=0 span_id=0 resource.service.name=mlx-app-design-py311-iris-9847-uxbwnvj2-2prlf9glng trace_sampled=False] - ########### log effective level is now: DEBUG 2024-08-28 06:46:09,354 DEBUG [root] [token_interceptor.py:55] [trace_id=0 span_id=0 resource.service.name=mlx-app-design-py311-iris-9847-uxbwnvj2-2prlf9glng trace_sampled=False] - ENABLE_SECURITY_FILTER:True for gRPC 2024-08-28 06:46:09,530 INFO [seldon_core.imports_helper] [imports_helper.py:39] [trace_id=0 span_id=0 resource.service.name=mlx-app-design-py311-iris-9847-uxbwnvj2-2prlf9glng trace_sampled=False] - Tensorflow is not installed. If you want to use tftensor and Tensorflow's data types install tensorflow or install seldon_core as

$ pip install seldon_core[tensorflow]

or

$ pip install seldon_core[all]

Traceback (most recent call last): File "/MLX_PYTHON/virtual_env/bin/seldon-core-microservice", line 5, in from seldon_core.microservice import main File "/microservice/python/seldon_core/microservice.py", line 15, in from seldon_core import wrapper as seldon_microservice File "/microservice/python/seldon_core/wrapper.py", line 13, in import seldon_core.seldon_methods File "/microservice/python/seldon_core/seldon_methods.py", line 21, in from seldon_core.proto import prediction_pb2 File "/microservice/python/seldon_core/proto/prediction_pb2.py", line 17, in from seldon_core.proto.tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensorpb2 File "/microservice/python/seldon_core/proto/tensorflow/core/framework/tensor_pb2.py", line 14, in from . import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resourcehandle__pb2 File "/microservice/python/seldon_core/proto/tensorflow/core/framework/resource_handle_pb2.py", line 14, in from . import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensorshapepb2 File "/microservice/python/seldon_core/proto/tensorflow/core/framework/tensor_shape_pb2.py", line 36, in _descriptor.FieldDescriptor( File "/otel-auto-instrumentation-python/google/protobuf/descriptor.py", line 553, in new _message.Message._CheckCalledFromGeneratedFile() TypeError: Descriptors cannot be created directly. If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0. If you cannot immediately regenerate your protos, some other possible workarounds are:

Downgrade the protobuf package to 3.20.x or lower.
Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates 2024-08-28 06:46:09,545 DEBUG [urllib3.connectionpool] [connectionpool.py:243] [trace_id=0 span_id=0 resource.service.name=mlx-app-design-py311-iris-9847-uxbwnvj2-2prlf9glng trace_sampled=False] - Starting new HTTP connection (1): obs-gateway-collector.fico-platform-observability.svc.cluster.local:4318 2024-08-28 06:46:09,558 DEBUG [urllib3.connectionpool] [connectionpool.py:546] [trace_id=0 span_id=0 resource.service.name=mlx-app-design-py311-iris-9847-uxbwnvj2-2prlf9glng trace_sampled=False] - http://obs-gateway-collector.fico-platform-observability.svc.cluster.local:4318 "POST /v1/metrics HTTP/11" 200 2`

Troubleshooting: We attempted to downgrade the Protobuf version by specifying the required version in the requirements.txt file, but this didn’t resolve the issue. We also tested with different Python versions but consistently encountered the same error.

Note: We discovered that without OpenTelemetry (OTEL), the application uses Protobuf version 3.20.3, but with OTEL, it defaults to version 4.25.4, which is not expected to be compatible with the Seldon Core base image. Why is OTEL enforcing its own Protobuf version?

Steps to Reproduce

Create a Flask application (v2.2.5) using the base image seldonio/seldon-core-s2i-python3:1.17.0 (https://hub.docker.com/r/seldonio/seldon-core-s2i-python3/tags) and try to instrument it using the Python auto-instrumentation image v0.47b (https://github.com/open-telemetry/opentelemetry-python/releases/tag/v1.26.0).

Expected Result

OTEL should capture logs, metrics, and traces.

Actual Result

The application fails to start with the attached error. model-container-python-0.47b.log (1).zip

Additional context

No response

Would you like to implement a fix?

None

xrmx commented 5 days ago

Are you sure it's us? opentelemetry-proto has this in its requirements:

dependencies = [
  "protobuf>=3.19, < 5.0",
]

ssurbhi19 commented 4 days ago

Yes, but how is the protobuf version 4.25.4 when OTEL is enabled? Please check the list of dependencies below: @xrmx

With OTEL 47.0b, below is the list of dependencies:

[mlxuser@mlx-app-design-py311-iris-9847-uxbwnvj2-2prlf9glng-97c9977twp2m microservice]$ pip list Package Version Editable project location

appdynamics 24.7.0.6967 appdynamics-bindeps-linux-x64 24.6.0 appdynamics-proxysupport-linux-x64 11.72.3 appnope 0.1.4 asgiref 3.8.1 asttokens 2.4.1 attrs 24.2.0 autocommand 2.2.2 backcall 0.2.0 backports.tarfile 1.2.0 beautifulsoup4 4.12.3 bleach 6.1.0 blinker 1.8.2 boto3 1.34.151 botocore 1.34.155 cachetools 5.3.2 certifi 2024.7.4 cffi 1.17.0 charset-normalizer 3.3.2 click 8.0.4 cryptography 41.0.7 decorator 5.1.1 defusedxml 0.7.1 Deprecated 1.2.14 dill 0.3.8 docopt 0.6.2 executing 2.0.1 fastjsonschema 2.20.0 fico-security-util 3 Flask 2.2.5 Flask-Cors 3.0.10 Flask-OpenTracing 1.1.0 googleapis-common-protos 1.63.2 grpcio 1.64.1 grpcio-opentracing 1.1.4 grpcio-reflection 1.48.2 gunicorn 20.1.0 idna 3.7 importlib_metadata 8.0.0 importlib_resources 6.4.0 inflect 7.3.1 ipython 8.12.3 itsdangerous 2.2.0 jaeger-client 4.4.0 jaraco.context 5.3.0 jaraco.functools 4.0.1 jaraco.text 3.12.1 jedi 0.19.1 Jinja2 3.1.4 jmespath 1.0.1 joblib 1.4.2 jpmml_evaluator 0.10.3 JPype1 1.5.0 jsonschema 4.23.0 jsonschema-specifications 2023.12.1 MarkupSafe 2.1.5 matplotlib-inline 0.1.7 mistune 3.0.2 more-itertools 10.3.0 nbclient 0.10.0 nbconvert 7.16.4 nbformat 5.10.4 numpy 1.26.4 opentelemetry-api 1.26.0 opentelemetry-distro 0.47b0 opentelemetry-exporter-otlp-proto-common 1.26.0 opentelemetry-exporter-otlp-proto-http 1.26.0 opentelemetry-exporter-prometheus 0.47b0 opentelemetry-instrumentation 0.47b0 opentelemetry-instrumentation-aio-pika 0.47b0 opentelemetry-instrumentation-aiohttp-client 0.47b0 opentelemetry-instrumentation-aiohttp-server 0.47b0 opentelemetry-instrumentation-aiopg 0.47b0 opentelemetry-instrumentation-asgi 0.47b0 opentelemetry-instrumentation-asyncio 0.47b0 opentelemetry-instrumentation-asyncpg 0.47b0 opentelemetry-instrumentation-boto 0.47b0 opentelemetry-instrumentation-boto3sqs 0.47b0 opentelemetry-instrumentation-botocore 0.47b0 opentelemetry-instrumentation-cassandra 0.47b0 opentelemetry-instrumentation-celery 0.47b0 opentelemetry-instrumentation-confluent-kafka 0.47b0 opentelemetry-instrumentation-dbapi 0.47b0 opentelemetry-instrumentation-django 0.47b0 opentelemetry-instrumentation-elasticsearch 0.47b0 opentelemetry-instrumentation-falcon 0.47b0 opentelemetry-instrumentation-fastapi 0.47b0 opentelemetry-instrumentation-flask 0.47b0 opentelemetry-instrumentation-grpc 0.47b0 opentelemetry-instrumentation-httpx 0.47b0 opentelemetry-instrumentation-jinja2 0.47b0 opentelemetry-instrumentation-kafka-python 0.47b0 opentelemetry-instrumentation-logging 0.47b0 opentelemetry-instrumentation-mysql 0.47b0 opentelemetry-instrumentation-mysqlclient 0.47b0 opentelemetry-instrumentation-pika 0.47b0 opentelemetry-instrumentation-psycopg 0.47b0 opentelemetry-instrumentation-psycopg2 0.47b0 opentelemetry-instrumentation-pymemcache 0.47b0 opentelemetry-instrumentation-pymongo 0.47b0 opentelemetry-instrumentation-pymysql 0.47b0 opentelemetry-instrumentation-pyramid 0.47b0 opentelemetry-instrumentation-redis 0.47b0 opentelemetry-instrumentation-remoulade 0.47b0 opentelemetry-instrumentation-requests 0.47b0 opentelemetry-instrumentation-sqlalchemy 0.47b0 opentelemetry-instrumentation-sqlite3 0.47b0 opentelemetry-instrumentation-starlette 0.47b0 opentelemetry-instrumentation-system-metrics 0.47b0 opentelemetry-instrumentation-threading 0.47b0 opentelemetry-instrumentation-tornado 0.47b0 opentelemetry-instrumentation-tortoiseorm 0.47b0 opentelemetry-instrumentation-urllib 0.47b0 opentelemetry-instrumentation-urllib3 0.47b0 opentelemetry-instrumentation-wsgi 0.47b0 opentelemetry-propagator-aws-xray 1.0.1 opentelemetry-propagator-b3 1.26.0 opentelemetry-propagator-jaeger 1.26.0 opentelemetry-propagator-ot-trace 0.47b0 opentelemetry-proto 1.26.0 opentelemetry-sdk 1.26.0 opentelemetry-semantic-conventions 0.47b0 opentelemetry-test-utils 0.47b0 opentelemetry-util-http 0.47b0 opentracing 2.4.0 ordered-set 4.1.0 packaging 24.1 pandas 2.1.4 pandocfilters 1.5.1 parso 0.8.4 pexpect 4.9.0 pickleshare 0.7.5 pip 24.0 pipdeptree 2.16.2 pipreqs 0.5.0 platformdirs 4.2.2 prometheus_client 0.20.0 prompt_toolkit 3.0.47 protobuf 4.25.4 psutil 6.0.0 ptyprocess 0.7.0 pure_eval 0.2.3 py4j 0.10.9.7 pycparser 2.22 Pygments 2.18.0 pyjnius 1.6.1 PyJWT 2.8.0 pyrsistent 0.20.0 python-dateutil 2.9.0.post0 pytz 2024.1 PyYAML 5.4.1 pyzmq 26.1.0 referencing 0.35.1 requests 2.32.3 rpds-py 0.20.0 s3transfer 0.10.2 scikit-learn 1.5.1 scipy 1.14.0 seldon-core 1.17.1 /microservice/python setuptools 72.1.0 six 1.16.0 soupsieve 2.5 stack-data 0.6.3 threadloop 1.0.2 threadpoolctl 3.5.0 thrift 0.20.0 tinycss2 1.3.0 tomli 2.0.1 tornado 6.4.1 traitlets 5.14.3 typeguard 4.3.0 typing_extensions 4.12.2 tzdata 2024.1 urllib3 2.2.2 wcwidth 0.2.13 webencodings 0.5.1 Werkzeug 2.2.3 wheel 0.44.0 wrapt 1.16.0 yarg 0.1.9 zipp 3.19.2

Without OTEL enabled:

[mlxuser@mlx-app-design-py311-iris-9847-uxbwnvj2-2prlf9glng-7bdd9f5tv52x microservice]$ pip list Package Version Editable project location

appdynamics 24.7.0.6967 appdynamics-bindeps-linux-x64 24.6.0 appdynamics-proxysupport-linux-x64 11.72.3 appnope 0.1.4 asttokens 2.4.1 attrs 24.2.0 backcall 0.2.0 beautifulsoup4 4.12.3 bleach 6.1.0 blinker 1.8.2 boto3 1.34.151 botocore 1.34.155 cachetools 5.3.2 certifi 2024.7.4 cffi 1.17.0 charset-normalizer 3.3.2 click 8.0.4 cryptography 41.0.7 decorator 5.1.1 defusedxml 0.7.1 dill 0.3.8 docopt 0.6.2 executing 2.0.1 fastjsonschema 2.20.0 fico-security-util 3 Flask 2.2.5 Flask-Cors 3.0.10 Flask-OpenTracing 1.1.0 grpcio 1.64.1 grpcio-opentracing 1.1.4 grpcio-reflection 1.48.2 gunicorn 20.1.0 idna 3.7 ipython 8.12.3 itsdangerous 2.2.0 jaeger-client 4.4.0 jedi 0.19.1 Jinja2 3.1.4 jmespath 1.0.1 joblib 1.4.2 jpmml_evaluator 0.10.3 JPype1 1.5.0 jsonschema 4.23.0 jsonschema-specifications 2023.12.1 MarkupSafe 2.1.5 matplotlib-inline 0.1.7 mistune 3.0.2 nbclient 0.10.0 nbconvert 7.16.4 nbformat 5.10.4 numpy 1.26.4 opentracing 2.4.0 packaging 24.1 pandas 2.1.4 pandocfilters 1.5.1 parso 0.8.4 pexpect 4.9.0 pickleshare 0.7.5 pip 24.0 pipdeptree 2.16.2 pipreqs 0.5.0 platformdirs 4.2.2 prometheus-client 0.8.0 prompt_toolkit 3.0.47 protobuf 3.20.3 psutil 6.0.0 ptyprocess 0.7.0 pure_eval 0.2.3 py4j 0.10.9.7 pycparser 2.22 Pygments 2.18.0 pyjnius 1.6.1 PyJWT 2.8.0 pyrsistent 0.20.0 python-dateutil 2.9.0.post0 pytz 2024.1 PyYAML 5.4.1 pyzmq 26.1.0 referencing 0.35.1 requests 2.32.3 rpds-py 0.20.0 s3transfer 0.10.2 scikit-learn 1.5.1 scipy 1.14.0 seldon-core 1.17.1 /microservice/python setuptools 69.5.1 six 1.16.0 soupsieve 2.5 stack-data 0.6.3 threadloop 1.0.2 threadpoolctl 3.5.0 thrift 0.20.0 tinycss2 1.3.0 tornado 6.4.1 traitlets 5.14.3 tzdata 2024.1 urllib3 2.2.2 wcwidth 0.2.13 webencodings 0.5.1 Werkzeug 2.2.3 wheel 0.44.0 yarg 0.1.9

ssurbhi19 commented 2 days ago

Hey @xrmx , Could you kindly assist me with this issue?

open-telemetry / opentelemetry-python