search

open-telemetry / opentelemetry-python

OpenTelemetry Python API and SDK
https://opentelemetry.io
Apache License 2.0
1.67k stars 570 forks source link

Unexpected behaviour observed after integrating with Otel #3860

Open anurag2994 opened 2 months ago

anurag2994 commented 2 months ago

Describe your environment Describe any aspect of your environment relevant to the problem, including your Python version, platform, version numbers of installed dependencies, information about your cloud hosting provider, etc. If you're reporting a problem with a specific version of a library in this repo, please check whether the problem has been fixed on main.

  1. Our service runs in docker.
  2. We use falcon server with gunicorn integration.
  3. We also use gevent patching for faster and non-blocking IO.
  4. We are using below versions of Otel:
  5. opentelemetry-api 1.24.0 opentelemetry-distro 0.45b0 opentelemetry-exporter-otlp 1.24.0 opentelemetry-exporter-otlp-proto-common 1.24.0 opentelemetry-exporter-otlp-proto-grpc 1.24.0 opentelemetry-exporter-otlp-proto-http 1.24.0 opentelemetry-instrumentation 0.45b0 opentelemetry-instrumentation-asgi 0.45b0 opentelemetry-instrumentation-asyncio 0.45b0 opentelemetry-instrumentation-aws-lambda 0.45b0 opentelemetry-instrumentation-boto 0.45b0 opentelemetry-instrumentation-boto3sqs 0.45b0 opentelemetry-instrumentation-botocore 0.45b0 opentelemetry-instrumentation-dbapi 0.45b0 opentelemetry-instrumentation-falcon 0.45b0 opentelemetry-instrumentation-grpc 0.45b0 opentelemetry-instrumentation-logging 0.45b0 opentelemetry-instrumentation-pymemcache 0.45b0 opentelemetry-instrumentation-pymysql 0.45b0 opentelemetry-instrumentation-requests 0.45b0 opentelemetry-instrumentation-sqlite3 0.45b0 opentelemetry-instrumentation-system-metrics 0.45b0 opentelemetry-instrumentation-urllib 0.45b0 opentelemetry-instrumentation-urllib3 0.45b0 opentelemetry-instrumentation-wsgi 0.45b0 opentelemetry-propagator-aws-xray 1.0.1 opentelemetry-proto 1.24.0 opentelemetry-sdk 1.24.0 opentelemetry-semantic-conventions 0.45b0 opentelemetry-test-utils 0.45b0 opentelemetry-util-http 0.45b0

Steps to reproduce Describe exactly how to reproduce the error. Include a code sample if applicable.

  1. Instrument falcon server in gunicorn master before server startup.
  2. Initialize Otel in post init worker.
  3. On a very higher load we are observing below issue:
  4. Our service made a db query Q1 and query Q2 at the same time. The results of Q1 and Q2 are getting interchanged. Suspect over here is that the db connection is somehow getting shared. What is the expected behavior? What did you expect to see?
  5. No db connection sharing should happen. What is the actual behavior? What did you see instead?
  6. We are suspecting db connection sharing. Additional context Add any other context about the problem here.