search

DataDog / dd-trace-py

Datadog Python APM Client
https://ddtrace.readthedocs.io/
Other
542 stars 411 forks source link

ddtrace.internal.writer.writer: failed to send error #10664

Open Apakottur opened 3 weeks ago

Apakottur commented 3 weeks ago

Summary of problem

We are getting the following error on our production environment:

ddtrace.internal.writer.writer: failed to send, dropping 43 traces to intake at http://localhost:8126/v0.5/traces after 3 retries, 52 additional messages skipped

Which version of dd-trace-py are you using?

2.12.2

Which version of pip are you using?

pip 24.2 python 3.12.5

Which libraries and their versions are you using?

`pip freeze` aiobotocore==2.15.0 aiohappyeyeballs==2.4.0 aiohttp==3.10.5 aioitertools==0.12.0 aiosignal==1.3.1 amplitude-analytics==1.1.4 annotated-types==0.7.0 anyio==4.4.0 argcomplete==3.4.0 asgi-lifespan==2.1.0 asyncpg==0.29.0 asyncpg-stubs==0.29.1 attrs==24.2.0 aws-export-credentials==0.18.0 boto3==1.35.16 botocore==1.35.16 botocore-stubs==1.35.17 bytecode==0.15.1 CacheControl==0.14.0 cachetools==5.5.0 cattrs==23.2.3 certifi==2024.8.30 cffi==1.17.1 cfgv==3.4.0 charset-normalizer==3.3.2 click==8.1.7 coverage==7.6.1 cryptography==43.0.1 ddsketch==3.0.1 ddtrace==2.12.2 Deprecated==1.2.14 distlib==0.3.8 dnspython==2.6.1 email_validator==2.2.0 emoji==2.12.1 envier==0.5.2 execnet==2.1.1 filelock==3.15.4 firebase-admin==6.5.0 freezegun==1.5.1 frozenlist==1.4.1 GeoAlchemy2==0.15.2 geographiclib==2.0 geopandas==1.0.1 geopy==2.4.1 google-api-core==2.19.2 google-api-python-client==2.145.0 google-auth==2.34.0 google-auth-httplib2==0.2.0 google-cloud-core==2.4.1 google-cloud-firestore==2.18.0 google-cloud-storage==2.18.2 google-crc32c==1.6.0 google-resumable-media==2.7.2 googleapis-common-protos==1.65.0 greenlet==3.1.0 grpcio==1.66.1 grpcio-status==1.66.1 h11==0.14.0 h3==3.7.7 httpcore==1.0.5 httplib2==0.22.0 httptools==0.6.1 httpx==0.27.2 identify==2.6.0 idna==3.8 importlib_metadata==8.4.0 iniconfig==2.0.0 Jinja2==3.1.4 jmespath==1.0.1 jsonschema==4.23.0 jsonschema-path==0.3.3 jsonschema-specifications==2023.12.1 lazy-object-proxy==1.10.0 loguru==0.7.2 markdown-it-py==3.0.0 MarkupSafe==2.1.5 mdurl==0.1.2 moto==5.0.14 msgpack==1.1.0 multidict==6.1.0 mypy==1.11.2 mypy-extensions==1.0.0 nodeenv==1.9.1 numpy==2.0.2 openapi-schema-validator==0.6.2 openapi-spec-validator==0.7.1 opentelemetry-api==1.27.0 packaging==24.1 pandas==2.2.2 pandas-stubs==2.2.2.240909 pathable==0.4.3 pgpasslib==1.1.0 phonenumbers==8.13.45 platformdirs==4.2.2 pluggy==1.5.0 pre-commit==3.8.0 proto-plus==1.24.0 protobuf==5.28.1 pyasn1==0.6.1 pyasn1_modules==0.4.1 pycparser==2.22 pydantic==2.9.1 pydantic-settings==2.5.2 pydantic_core==2.23.3 Pygments==2.18.0 pyinstrument==4.7.3 PyJWT==2.9.0 pyogrio==0.9.0 pyparsing==3.1.4 pyproj==3.6.1 pyright==1.1.380 pytest==8.3.3 pytest-asyncio==0.24.0 pytest-cov==5.0.0 pytest-httpx==0.30.0 pytest-mock==3.14.0 pytest-sugar==1.0.0 pytest-xdist==3.6.1 python-dateutil==2.9.0.post0 python-dotenv==1.0.1 pytz==2024.2 PyYAML==6.0.2 redis==5.0.8 referencing==0.35.1 requests==2.32.3 responses==0.25.3 rfc3339-validator==0.1.4 rich==13.8.1 rpds-py==0.20.0 rsa==4.9 s3transfer==0.10.2 setuptools==74.1.2 shapely==2.0.6 shellingham==1.5.4 shpyx==0.0.29 six==1.16.0 sniffio==1.3.1 SQLAlchemy==2.0.34 starlette==0.38.5 tenacity==9.0.0 termcolor==2.4.0 timezonefinder==6.5.2 typeguard==4.3.0 typer==0.12.5 types-aiobotocore==2.15.0 types-aiobotocore-personalize-runtime==2.15.0 types-aiobotocore-rekognition==2.15.0 types-aiobotocore-s3==2.15.0 types-aiobotocore-sqs==2.15.0 types-aiobotocore-stepfunctions==2.15.0 types-awscrt==0.21.5 types-python-dateutil==2.9.0.20240906 types-pytz==2024.1.0.20240417 types-shapely==2.0.0.20240820 typing_extensions==4.12.2 tzdata==2024.1 uritemplate==4.1.1 urllib3==2.2.3 uvicorn==0.30.6 uvloop==0.20.0 virtualenv==20.26.3 watchfiles==0.24.0 websockets==13.0.1 Werkzeug==3.0.4 wrapt==1.16.0 xmltodict==0.13.0 yarl==1.11.1 zipp==3.20.1

How can we reproduce your problem?

I'm not sure, this just starts hapenning on our production environment after running for a few hours.

What is the result that you get?

We are getting the error periodically from our production ddtrace instances: ddtrace_error

What is the result that you expected?

No errors from ddtrace.

How can we debug this further?

Apakottur commented 5 days ago

We ended up pinning ddtrace to 2.11.2 and we didn't have the error since (so more than two weeks now), so it's something in the newer versions.