search

DataDog / dd-trace-py

Datadog Python APM Client
https://ddtrace.readthedocs.io/
Other
551 stars 413 forks source link

ddtrace.internal.remoteconfig.client: Cannot assign requested address #5034

Closed lucas03 closed 1 year ago

lucas03 commented 1 year ago

Summary of problem

We started to see these logs after we upgraded to a newer version of datadog.

Specifically in our services it's 5M logs in past day. I don't think there is any impact on logs/traces, but these warnings are noisy and storing/indexing them costs something.

This is traceback:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/remoteconfig/client.py", line 425, in request
    response = self._send_request(payload)
  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/remoteconfig/client.py", line 266, in _send_request
    self._conn.request("POST", "v0.7/config", payload, self._headers)
  File "/usr/local/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/usr/local/lib/python3.10/http/client.py", line 941, in connect
    self.sock = self._create_connection(
  File "/usr/local/lib/python3.10/site-packages/gevent/socket.py", line 117, in create_connection
    sock.connect(sa)
  File "/usr/local/lib/python3.10/site-packages/gevent/_socketcommon.py", line 628, in connect
    raise _SocketError(result, strerror(result))
OSError: [Errno 99] Cannot assign requested address

Which version of dd-trace-py are you using?

ddtrace==1.7.2

Which version of pip are you using?

pip 21.3.1

Which libraries and their versions are you using?

`pip freeze` aioredis==2.0.1 alembic==1.8.1 amqp==5.1.1 arrow==1.2.3 asn1crypto==1.5.1 astroid==2.12.12 async-timeout==4.0.2 attrs==22.1.0 Babel==2.10.3 beautifulsoup4==4.11.1 billiard==3.6.4.0 black==22.10.0 bleach==5.0.1 blinker==1.5 boto3==1.25.3 botocore==1.28.3 bottle==0.12.23 brazilnum==0.8.8 build==0.9.0 bytecode==0.13.0 cached-property==1.5.2 cachetools==5.2.0 cattrs==22.2.0 celery==5.2.7 Cerberus==1.3.4 certifi==2022.12.7 cffi==1.15.1 cfgv==3.3.1 chardet==4.0.0 charset-normalizer==2.1.1 click==8.1.3 click-didyoumean==0.3.0 click-plugins==1.1.1 click-repl==0.2.0 clickclick==20.10.2 colorama==0.4.6 connexion==2.14.1 coverage==6.5.0 cryptography==39.0.0 datadog==0.44.0 ddsketch==2.0.4 ddtrace==1.5.3 defusedxml==0.7.1 Deprecated==1.2.13 dictalchemy==0.1.2.7 dill==0.3.6 distlib==0.3.6 dnspython==2.2.1 email-validator==1.3.0 envier==0.4.0 envparse==0.2.0 exceptiongroup==1.0.0 execnet==1.9.0 factory-boy==3.0.1 Faker==15.1.1 fakeredis==1.10.0 filelock==3.8.0 filetype==1.2.0 Flask==2.2.2 freezegun==0.3.15 future==0.18.2 geographiclib==1.52 geopy==2.2.0 gevent==22.10.1 google-api-core==2.10.2 google-auth==2.13.0 google-cloud-core==2.3.2 google-cloud-pubsub==2.13.10 google-cloud-storage==2.5.0 google-crc32c==1.5.0 google-resumable-media==2.4.0 googleapis-common-protos==1.56.4 greenlet==1.1.3.post0 grpc-google-iam-v1==0.12.4 grpcio==1.50.0 grpcio-status==1.50.0 gunicorn==20.1.0 hiredis==2.0.0 identify==2.5.6 idna==3.4 importlib-metadata==5.0.0 importlib-resources==5.10.2 inflection==0.5.1 iniconfig==1.1.1 iso3166==2.1.1 iso4217==1.11.20220401 iso639==0.1.4 isodate==0.6.1 isort==5.10.1 itsdangerous==2.1.2 Jinja2==3.1.2 jmespath==1.0.1 jsonschema==4.16.0 jsonschema-spec==0.1.2 kombu==5.2.4 language-tags==1.1.0 lazy-object-proxy==1.9.0 lxml==4.9.1 lz4==4.0.2 Mako==1.2.3 MarkupSafe==2.1.1 marshmallow==3.18.0 mccabe==0.7.0 mock==4.0.3 money==1.3.0 monotonic==1.6 mypy==0.982 mypy-extensions==0.4.3 mypy-nonfloat-decimal==0.1.7 netaddr==0.8.0 nodeenv==1.7.0 numpy==1.24.0 openapi-schema-validator==0.3.4 openapi-spec-validator==0.5.1 packaging==21.3 pathable==0.4.3 pathlib2==2.3.7.post1 pathspec==0.10.2 pep517==0.13.0 pg8000==1.29.3 phonenumbers==8.12.57 pip-tools==6.9.0 platformdirs==2.5.2 pluggy==1.0.0 prance==0.22.11.4.0 pre-commit==2.20.0 prompt-toolkit==3.0.31 proto-plus==1.22.1 protobuf==4.21.9 psycogreen==1.0.2 psycopg2-binary==2.9.5 py==1.11.0 pyaml==21.10.1 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycountry==22.3.5 pycparser==2.21 pydantic==1.10.2 pydantic-factories==1.13.0 PyJWT==2.6.0 pylint==2.15.5 pyparsing==3.0.9 pyrsistent==0.18.1 pytest==7.2.0 pytest-celery==0.0.0 pytest-cov==4.0.0 pytest-factoryboy==2.5.0 pytest-forked==1.4.0 pytest-mock==3.10.0 pytest-randomly==3.12.0 pytest-timeout==2.1.0 pytest-xdist==3.0.2 python-dateutil==2.8.2 python-magic==0.4.27 pytz==2022.5 PyYAML==6.0 raven==6.10.0 redis==4.3.4 request-session==0.15.0 requests==2.28.1 requests-file==1.5.1 requests-mock==1.10.0 requests-toolbelt==0.10.1 rsa==4.9 ruamel.yaml==0.17.21 ruamel.yaml.clib==0.2.7 s3transfer==0.6.0 schwifty==2022.9.0 scramp==1.4.3 semver==2.13.0 sentry-sdk==1.10.1 setproctitle==1.3.2 simplejson==3.17.6 six==1.16.0 sortedcontainers==2.4.0 soupsieve==2.3.2.post1 sql-metadata==2.6.0 SQLAlchemy==1.4.42 sqlalchemy-stubs==0.4 sqlparse==0.4.3 structlog==22.1.0 swagger-ui-bundle==0.0.9 tenacity==8.1.0 testing.common.database==2.0.3 testing.postgresql==1.3.0 tinys3==0.1.12 toml==0.10.2 tomli==2.0.1 tomlkit==0.11.6 types-bleach==5.0.3.1 types-cachetools==5.2.1 types-certifi==2021.10.8.3 types-pytz==2022.5.0.0 types-PyYAML==6.0.12.1 types-redis==4.3.21.3 types-requests==2.28.11.2 types-simplejson==3.17.7.1 types-urllib3==1.26.25.1 typing_extensions==4.4.0 urllib3==1.26.12 vine==5.0.0 virtualenv==20.16.5 waitress==2.1.2 wcwidth==0.2.5 webencodings==0.5.1 WebOb==1.8.7 WebTest==3.0.0 Werkzeug==2.2.2 wrapt==1.14.1 xeger==0.3.5 XlsxWriter==3.0.3 xmltodict==0.13.0 zeep==4.1.0 zipp==3.10.0 zope.event==4.5.0 zope.interface==5.5.0 zstandard==0.18.0

How can we reproduce your problem?

I guess using latest version?

What is the result that you get?

logger info with errors

What is the result that you expected?

logs related to remoteconfig won't be produced

brettlangdon commented 1 year ago

hey @lucas03 thank you for opening this issue, and apologies you are seeing a ton of logs like this.

We have created and merged #4969, which may help reduce the burden of these log lines.

The change includes, not including a traceback, because it isn't useful for you the user (it isn't really that useful for our debugging either), and fixing the logger to use a rate limited logger.

By default all of our modules use a rate limited logger so we will never log the same message more than once every 60 seconds. This module originally did not use it, and we updated to fix that.

This change is merged into our 1.x branch, and will be included in the 1.9.0 release (ETA tbd, but probably no more than 2-3 weeks from now).

Do you think these changes would help alleviate your concerns?

lucas03 commented 1 year ago

Hey @brettlangdon, thanks for swift reply. And appologies I didn't link the merge request, I actually found it a day or two ago.

So of logger is rate limited, it will still produce logs but in fewer numbers? Meaning, if we have 100 containers, I can expect at most 100*60 logs per hour?

What is the reason for the log? Can I just disable it with DD_REMOTE_CONFIGURATION_ENABLED=false

brettlangdon commented 1 year ago

So of logger is rate limited, it will still produce logs but in fewer numbers? Meaning, if we have 100 containers, I can expect at most 100*60 logs per hour?

Correct. If the log occurs multiples times per-minute, the next log allowed through with have a message like , X additional messages skipped or something (sorry, being lazy and not looking in the code)

We introduced this in case any logs we produce happen in a hot path, we don't want to spam a user's logs in case something that happens on every request occurs, or occurs in a loop, etc. Better safe than sorry.

What is the reason for the log?

It likely means the version of your trace agent does not support remote configuration functionality. So it really just means "if you try to use remote config, it won't work".

Can I just disable it with DD_REMOTE_CONFIGURATION_ENABLED=false

Yes, disabling remote config via that env var should prevent the log lines from happening.

lucas03 commented 1 year ago

okay, I guess we can resolve this. Thanks!