search

DataDog / dd-trace-py

Datadog Python APM Client
https://ddtrace.readthedocs.io/
Other
551 stars 412 forks source link

TypeError: '<' not supported between instances of 'int' and 'NoneType' #3019

Closed Apakottur closed 2 years ago

Apakottur commented 2 years ago

Thanks for this awesome integration with Python.

We are using Sentry and Datadog (with profiling enabled) to monitor our system. Occasionally, we get Datadog errors, like the following one:

sentry_sdk.integrations.logging:sentry_patched_callhandlers:86 - Unexpected error while exporting events.

Please report this bug to https://github.com/DataDog/dd-trace-py/issuesTraceback (most recent call last): |

File "/usr/local/lib/python3.10/threading.py", line 966, in _bootstrap

self._bootstrap_inner()

<function Thread._bootstrap_inner at 0x7fe4ded153f0>

<PeriodicThread(ddtrace.profiling.scheduler:Scheduler, started daemon 140620893931264)>

File "/usr/local/lib/python3.10/threading.py", line 1009, in _bootstrap_inner

self.run()

<function PeriodicThread.run at 0x7fe4dd2312d0>

<PeriodicThread(ddtrace.profiling.scheduler:Scheduler, started daemon 140620893931264)>

File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/periodic.py", line 70, in run

self._target()

<bound method Scheduler.periodic of Scheduler(status=<ServiceStatus.RUNNING: 'running'>, recorder=Recorder(default_max_events...

<PeriodicThread(ddtrace.profiling.scheduler:Scheduler, started daemon 140620893931264)>

File "/usr/local/lib/python3.10/site-packages/ddtrace/profiling/scheduler.py", line 65, in periodic

self.flush()

<function Scheduler.flush at 0x7fe4dd28bb50>

Scheduler(status=<ServiceStatus.RUNNING: 'running'>, recorder=Recorder(default_max_events=32768, max_events={<class 'ddtrace....

File "/usr/local/lib/python3.10/site-packages/ddtrace/profiling/scheduler.py", line 53, in flush

exp.export(events, start, self._last_export)

Scheduler(status=<ServiceStatus.RUNNING: 'running'>, recorder=Recorder(default_max_events=32768, max_events={<class 'ddtrace....

{<class 'ddtrace.profiling.collector.stack.StackSampleEvent'>: deque([StackSampleEvent(timestamp=1637081996232555786, samplin...

<function PprofHTTPExporter.export at 0x7fe4dbf95630>

PprofHTTPExporter(endpoint='http://localhost:8126', api_key=None, timeout=10.0, service=None, env=None, version=None, tags={'...

File "/usr/local/lib/python3.10/site-packages/ddtrace/profiling/exporter/http.py", line 150, in export

profile = super(PprofHTTPExporter, self).export(events, start_time_ns, end_time_ns)

{<class 'ddtrace.profiling.collector.stack.StackSampleEvent'>: deque([StackSampleEvent(timestamp=1637081996232555786, samplin...

PprofHTTPExporter(endpoint='http://localhost:8126', api_key=None, timeout=10.0, service=None, env=None, version=None, tags={'...

<class 'ddtrace.profiling.exporter.http.PprofHTTPExporter'>

File "ddtrace/profiling/exporter/pprof.pyx", line 535, in ddtrace.profiling.exporter.pprof.PprofExporter.export

File "ddtrace/profiling/exporter/pprof.pyx", line 406, in ddtrace.profiling.exporter.pprof.PprofExporter._group_stack_events

TypeError: '<' not supported between instances of 'int' and 'NoneType'

I don't know if it's related to Sentry or not, and I don't have anything except the traceback.

This happens rarely, and we are able to use both tools without issues.

Which version of dd-trace-py are you using?

ddtrace = "0.55.4"

Which version of pip are you using?

pip 21.2.4 from /usr/local/lib/python3.10/site-packages/pip (python 3.10)

Which version of the libraries are you using?

aiobotocore==2.0.0 aiodns==3.0.0 aiohttp==3.8.1 aioitertools==0.8.0 aioredis==1.3.1 aiosignal==1.2.0 aiosmtplib==1.1.6 alembic==1.7.5 anyio==3.3.4 arq==0.22 asgiref==3.4.1 async-timeout==4.0.1 asyncpg==0.25.0 attrs==21.2.0 Babel==2.9.1 backports.entry-points-selectable==1.1.1 bcrypt==3.2.0 boto3==1.19.8 boto3-stubs==1.20.8 botocore==1.22.8 botocore-stubs==1.23.8 CacheControl==0.12.10 cached-property==1.5.2 cachy==0.3.0 certifi==2021.10.8 cffi==1.15.0 charset-normalizer==2.0.7 cleo==0.8.1 click==8.0.3 clikit==0.6.2 coverage==6.1.2 crashtest==0.3.1 cryptography==35.0.0 ddtrace==0.55.4 distlib==0.3.3 dnspython==2.1.0 email-normalize==2.0.0 email-validator==1.1.3 execnet==1.9.0 filelock==3.3.2 freezegun==1.1.0 frozenlist==1.2.0 graphql-core==3.1.6 greenlet==1.1.2 gunicorn==20.1.0 h11==0.12.0 hiredis==2.0.0 html5lib==1.1 httpcore==0.14.3 httptools==0.2.0 httpx==0.21.1 idna==3.3 iniconfig==1.1.1 jeepney==0.7.1 Jinja2==3.0.3 jmespath==0.10.0 keyring==21.8.0 lockfile==0.12.2 loguru==0.5.3 Mako==1.1.6 MarkupSafe==2.0.1 migra==3.0.1621480950 more-itertools==8.11.0 moto==2.2.15 msgpack==1.0.2 multidict==5.2.0 mypy==0.910 mypy-extensions==0.4.3 packaging==21.3 passlib==1.7.4 pastel==0.2.1 pexpect==4.8.0 phonenumbers==8.12.37 pkginfo==1.7.1 platformdirs==2.4.0 pluggy==1.0.0 poetry==1.1.11 poetry-core==1.0.7 protobuf==3.19.1 psycopg2==2.9.2 ptyprocess==0.7.0 py==1.11.0 pycares==4.1.2 pycparser==2.21 pydantic==1.8.2 Pygments==2.10.0 PyJWT==2.3.0 pylev==1.4.0 pyparsing==3.0.6 pytest==6.2.5 pytest-asyncio==0.16.0 pytest-cov==3.0.0 pytest-forked==1.3.0 pytest-httpx==0.15.0 pytest-mock==3.6.1 pytest-timeout==2.0.1 pytest-xdist==2.4.0 python-dateutil==2.8.2 python-dotenv==0.19.2 python-multipart==0.0.5 pytz==2021.3 PyYAML==6.0 requests==2.26.0 requests-toolbelt==0.9.1 responses==0.16.0 rfc3986==1.5.0 s3transfer==0.5.0 schemainspect==3.0.1616029793 SecretStorage==3.3.1 sentinel==0.3.0 sentry-sdk==1.5.0 shellingham==1.4.0 shpyx==0.0.10 simplejson==3.17.6 six==1.16.0 sniffio==1.2.0 SQLAlchemy==1.4.27 sqlalchemy2-stubs==0.0.2a19 sqlbag==0.1.1617247075 starlette==0.17.1 strawberry-graphql==0.87.1 tenacity==8.0.1 toml==0.10.2 tomli==1.2.2 tomlkit==0.7.2 types-babel==2.9.1 types-python-dateutil==2.8.2 types-simplejson==3.17.1 typing-extensions==3.10.0.2 urllib3==1.26.7 uvicorn==0.15.0 uvloop==0.16.0 virtualenv==20.10.0 watchgod==0.7 webencodings==0.5.1 websockets==10.1 Werkzeug==2.0.2 wrapt==1.13.3 xmltodict==0.12.0 yarl==1.7.2

jd commented 2 years ago

Likely a duplicate of #2962 It should be fixed in the next release 🤞

Apakottur commented 2 years ago

Awesome, thanks for the response! Feel free to close this if its a duplicate.

jd commented 2 years ago

Thank you — closing. Feel free to reopen if it's not fixed in the next release!

Apakottur commented 2 years ago

I just had this error again on 0.56.1: ddtrace1

Should we reopen the issue? Can't do it myself.

jd commented 2 years ago

It's scheduled for 0.57 but I'll backport it for 0.56 in https://github.com/DataDog/dd-trace-py/pull/3062

Apakottur commented 2 years ago

@jd We just got this again on 0.57.3, should I open a new issue? ddtrace_1

adamos-kyriakou-wolt commented 2 years ago

I can verify we're still getting these errors on the latest v0.57.3 version as well. @Apakottur any progress?

Weirdly enough we only started getting these after introducing https://github.com/claws/aioprometheus for capturing metrics against our prometheus instance

Apakottur commented 2 years ago

@adamos-kyriakou-wolt Not too much, we keep getting this error and just ignore it for now. I also suspect that this has to do with other monitoring tools, like Sentry which we use in parallel.

One thing that I noticed is that we get this almost every time when we deploy our application (and the DD initialization code runs), but it also happens during the runtime of the application.

adamos-kyriakou-wolt commented 2 years ago

Same thing here @Apakottur, we're running both Sentry and DD over our service and get these errors during runtime. Let's hope @jd will have some news for us both 🙏