search

DataDog / dd-trace-py

Datadog Python APM Client
https://ddtrace.readthedocs.io/
Other
542 stars 411 forks source link

DDTrace is incompatible with pytest 8 #8220

Closed ravirahman closed 7 months ago

ravirahman commented 8 months ago

DDTrace is incompatible with pytest 8

Which version of dd-trace-py are you using?

2.5.1

Which version of pip are you using?

23.0.1

Which libraries and their versions are you using?

pytest 8.0.0

aiobotocore==2.11.1
aiohttp==3.9.2
aioitertools==0.11.0
aiomysql==0.2.0
aiosignal==1.3.1
annotated==types==0.6.0
asn1crypto==1.5.1
asttokens==2.4.1
async==timeout==4.0.3
attrs==23.2.0
backcall==0.2.0
backports.zoneinfo==0.2.1
boto3==1.34.27
botocore==1.34.27
bytecode==0.15.1
cachetools==5.3.2
cattrs==23.2.3
certifi==2023.11.17
cffi==1.16.0
charset==normalizer==3.3.2
cloudpickle==2.2.1
comm==0.2.1
cryptography==41.0.7
dataclasses==json==0.6.3
ddsketch==2.0.4
ddtrace==2.5.1
decorator==5.1.1
deprecated==1.2.14
duckdb==0.6.0
envier==0.5.1
exceptiongroup==1.2.0
execnet==2.0.2
executing==1.2.0
filelock==3.13.1
frozenlist==1.4.1
fsspec==2023.12.2
gcsfs==2023.12.2.post1
google==api==core==2.15.0
google==auth==2.27.0
google==auth==oauthlib==1.2.0
google==cloud==core==2.4.1
google==cloud==storage==2.14.0
google==crc32c==1.5.0
google==resumable==media==2.7.0
googleapis==common==protos==1.62.0
greenlet==3.0.3
idna==3.6
importlib==metadata==6.11.0
iniconfig==2.0.0
ipython==8.12.3
ipywidgets==8.1.1
isodate==0.6.1
jedi==0.19.1
jmespath==1.0.1
jupyterlab==widgets==3.0.9
levenshtein==0.24.0
markdown==it==py==3.0.0
marshmallow==3.20.2
matplotlib==inline==0.1.6
mdurl==0.1.2
multidict==6.0.4
mypy==extensions==1.0.0
numpy==1.24.4
oauthlib==3.2.2
opentelemetry==api==1.22.0
packaging==23.2
pandas==1.5.3
parso==0.8.3
pexpect==4.9.0
pickleshare==0.7.5
platformdirs==3.11.0
pluggy==1.4.0
polars==0.17.2
prompt==toolkit==3.0.43
protobuf==4.25.1
psycopg==3.1.17
psycopg==binary==3.1.17
psycopg2==2.9.9
ptyprocess==0.7.0
pure==eval==0.2.2
pyOpenSSL==23.3.0
pyarrow==14.0.2
pyarrow==hotfix==0.6
pyasn1==0.5.1
pyasn1==modules==0.3.0
pycparser==2.21
pydantic==2.6.0
pydantic==core==2.16.1
pygments==2.17.2
pyjwt==2.8.0
pymysql==1.1.0
pytest==8.0.0
pytest==xdist==3.5.0
python==dateutil==2.8.2
python==dotenv==1.0.1
pytz==2023.4
pyyaml==6.0.1
rapidfuzz==3.6.1
requests==2.31.0
requests==oauthlib==1.3.1
rich==13.7.0
rsa==4.9
s3fs==2023.12.2
s3transfer==0.10.0
six==1.16.0
snowflake==connector==python==3.5.0
snowflake==sqlalchemy==1.5.1
sortedcontainers==2.4.0
sqlalchemy==1.4.51
sqlglot==20.11.0
stack==data==0.6.3
tomli==2.0.1
tomlkit==0.12.3
tqdm==4.66.1
traitlets==5.14.1
typing==extensions==4.9.0
typing==inspect==0.9.0
tzdata==2023.4
urllib3==1.26.18
wcwidth==0.2.13
widgetsnbextension==4.0.9
wrapt==1.16.0
xmltodict==0.13.0
yarl==1.9.4
zipp==3.17.0

How can we reproduce your problem?

Run pytest with ddtrace and pytest 8

What is the result that you get?

[Datadog CI Visibility] INFO     ddtrace.internal.ci_visibility.utils:utils.py:141 logger setup complete
[Datadog CI Visibility] INFO     ddtrace.internal.ci_visibility.recorder:recorder.py:163 Datadog CI Visibility using agentless mode
[Datadog CI Visibility] WARNING  ddtrace.internal.ci_visibility.recorder:recorder.py:184 Intelligent Test Runner test skipping disabled by API
[Datadog CI Visibility] WARNING  ddtrace.internal.ci_visibility.recorder:recorder.py:191 CODEOWNERS file is not available
PLUGIN registered: <_pytest.config.PytestPluginManager object at 0x7f75ef1c0550>
PLUGIN registered: <_pytest.config.Config object at 0x7f75ef4bda90>
PLUGIN registered: <module '_pytest.mark' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/mark/__init__.py'>
PLUGIN registered: <module '_pytest.main' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/main.py'>
PLUGIN registered: <module '_pytest.runner' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/runner.py'>
PLUGIN registered: <module '_pytest.fixtures' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/fixtures.py'>
PLUGIN registered: <module '_pytest.helpconfig' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/helpconfig.py'>
PLUGIN registered: <module '_pytest.python' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/python.py'>
PLUGIN registered: <module '_pytest.terminal' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/terminal.py'>
PLUGIN registered: <module '_pytest.debugging' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/debugging.py'>
PLUGIN registered: <module '_pytest.unittest' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/unittest.py'>
PLUGIN registered: <module '_pytest.capture' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/capture.py'>
PLUGIN registered: <module '_pytest.skipping' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/skipping.py'>
PLUGIN registered: <module '_pytest.legacypath' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/legacypath.py'>
PLUGIN registered: <module '_pytest.tmpdir' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/tmpdir.py'>
PLUGIN registered: <module '_pytest.monkeypatch' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/monkeypatch.py'>
PLUGIN registered: <module '_pytest.recwarn' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/recwarn.py'>
PLUGIN registered: <module '_pytest.pastebin' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/pastebin.py'>
PLUGIN registered: <module '_pytest.nose' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/nose.py'>
PLUGIN registered: <module '_pytest.assertion' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/assertion/__init__.py'>
PLUGIN registered: <module '_pytest.junitxml' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/junitxml.py'>
PLUGIN registered: <module '_pytest.doctest' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/doctest.py'>
PLUGIN registered: <module '_pytest.cacheprovider' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/cacheprovider.py'>
PLUGIN registered: <module '_pytest.freeze_support' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/freeze_support.py'>
PLUGIN registered: <module '_pytest.setuponly' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/setuponly.py'>
PLUGIN registered: <module '_pytest.setupplan' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/setupplan.py'>
PLUGIN registered: <module '_pytest.stepwise' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/stepwise.py'>
PLUGIN registered: <module '_pytest.warnings' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/warnings.py'>
PLUGIN registered: <module '_pytest.logging' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/logging.py'>
PLUGIN registered: <module '_pytest.reports' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/reports.py'>
PLUGIN registered: <module '_pytest.python_path' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/python_path.py'>
PLUGIN registered: <module '_pytest.unraisableexception' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/unraisableexception.py'>
PLUGIN registered: <module '_pytest.threadexception' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/threadexception.py'>
PLUGIN registered: <module '_pytest.faulthandler' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/faulthandler.py'>
PLUGIN registered: <module 'ddtrace.contrib.pytest.plugin' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest/plugin.py'>
PLUGIN registered: <module 'ddtrace.contrib.pytest_bdd.plugin' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest_bdd/plugin.py'>
PLUGIN registered: <module 'ddtrace.contrib.pytest_benchmark.plugin' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest_benchmark/plugin.py'>
PLUGIN registered: <module 'xdist.plugin' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/plugin.py'>
PLUGIN registered: <module 'xdist.looponfail' from '/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/looponfail.py'>
PLUGIN registered: <CaptureManager _method='fd' _global_capturing=<MultiCapture out=<FDCapture 1 oldfd=7 _state='suspended' tmpfile=<_io.TextIOWrapper name="<_io.FileIO name=8 mode='rb+' closefd=True>" mode='r+' encoding='utf-8'>> err=<FDCapture 2 oldfd=9 _state='suspended' tmpfile=<_io.TextIOWrapper name="<_io.FileIO name=10 mode='rb+' closefd=True>" mode='r+' encoding='utf-8'>> in_=<FDCapture 0 oldfd=5 _state='started' tmpfile=<_io.TextIOWrapper name='/dev/null' mode='r' encoding='utf-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>
PLUGIN registered: <Session  exitstatus=<ExitCode.OK: 0> testsfailed=0 testscollected=0>
PLUGIN registered: <_pytest.cacheprovider.LFPlugin object at 0x7f75edc3ea30>
PLUGIN registered: <_pytest.cacheprovider.NFPlugin object at 0x7f75edc3ea00>
PLUGIN registered: <class '_pytest.legacypath.LegacyTmpdirPlugin'>
PLUGIN registered: <_pytest.terminal.TerminalReporter object at 0x7f75ed8c9970>
PLUGIN registered: <_pytest.logging.LoggingPlugin object at 0x7f75ed8c9f70>
PLUGIN registered: <xdist.dsession.TerminalDistReporter object at 0x7f75ec0f3e80>
PLUGIN registered: <xdist.dsession.DSession object at 0x7f75ec0f3340>
PLUGIN registered: <_pytest.fixtures.FixtureManager object at 0x7f75ec0f3e50>
============================= test session starts ==============================
platform linux -- Python 3.8.18, pytest-8.0.0, pluggy-1.4.0
using: pytest-8.0.0
setuptools registered plugins:
  ddtrace-2.5.1 at /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest/plugin.py
  ddtrace-2.5.1 at /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest_bdd/plugin.py
  ddtrace-2.5.1 at /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest_benchmark/plugin.py
  pytest-xdist-3.5.0 at /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/plugin.py
  pytest-xdist-3.5.0 at /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/looponfail.py
active plugins:
    140144499492176     : <_pytest.config.PytestPluginManager object at 0x7f75ef1c0550>
    pytestconfig        : <_pytest.config.Config object at 0x7f75ef4bda90>
    mark                : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/mark/__init__.py
    main                : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/main.py
    runner              : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/runner.py
[Datadog CI Visibility] INFO     ddtrace.internal.ci_visibility.utils:utils.py:141 logger setup complete
[Datadog CI Visibility] INFO     ddtrace.internal.ci_visibility.utils:utils.py:141 logger setup complete
[Datadog CI Visibility] INFO     ddtrace.internal.ci_visibility.recorder:recorder.py:163 Datadog CI Visibility using agentless mode
[Datadog CI Visibility] INFO     ddtrace.internal.ci_visibility.utils:utils.py:141 logger setup complete
[Datadog CI Visibility] INFO     ddtrace.internal.ci_visibility.recorder:recorder.py:163 Datadog CI Visibility using agentless mode
[Datadog CI Visibility] WARNING  ddtrace.internal.ci_visibility.recorder:recorder.py:184 Intelligent Test Runner test skipping disabled by API
[Datadog CI Visibility] WARNING  ddtrace.internal.ci_visibility.recorder:recorder.py:191 CODEOWNERS file is not available
[Datadog CI Visibility] INFO     ddtrace.internal.ci_visibility.recorder:recorder.py:163 Datadog CI Visibility using agentless mode
[Datadog CI Visibility] WARNING  ddtrace.internal.ci_visibility.recorder:recorder.py:184 Intelligent Test Runner test skipping disabled by API
[Datadog CI Visibility] WARNING  ddtrace.internal.ci_visibility.recorder:recorder.py:191 CODEOWNERS file is not available
[Datadog CI Visibility] INFO     ddtrace.internal.ci_visibility.utils:utils.py:141 logger setup complete
[Datadog CI Visibility] WARNING  ddtrace.internal.ci_visibility.recorder:recorder.py:184 Intelligent Test Runner test skipping disabled by API
[Datadog CI Visibility] WARNING  ddtrace.internal.ci_visibility.recorder:recorder.py:191 CODEOWNERS file is not available
[Datadog CI Visibility] INFO     ddtrace.internal.ci_visibility.recorder:recorder.py:163 Datadog CI Visibility using agentless mode
[Datadog CI Visibility] WARNING  ddtrace.internal.ci_visibility.recorder:recorder.py:184 Intelligent Test Runner test skipping disabled by API
[Datadog CI Visibility] WARNING  ddtrace.internal.ci_visibility.recorder:recorder.py:191 CODEOWNERS file is not available
    fixtures            : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/fixtures.py
    helpconfig          : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/helpconfig.py
    python              : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/python.py
    terminal            : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/terminal.py
    debugging           : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/debugging.py
    unittest            : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/unittest.py
    capture             : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/capture.py
    skipping            : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/skipping.py
    legacypath          : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/legacypath.py
    tmpdir              : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/tmpdir.py
    monkeypatch         : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/monkeypatch.py
    recwarn             : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/recwarn.py
    pastebin            : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/pastebin.py
    nose                : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/nose.py
    assertion           : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/assertion/__init__.py
    junitxml            : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/junitxml.py
    doctest             : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/doctest.py
    cacheprovider       : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/cacheprovider.py
    freeze_support      : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/freeze_support.py
    setuponly           : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/setuponly.py
    setupplan           : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/setupplan.py
    stepwise            : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/stepwise.py
    warnings            : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/warnings.py
    logging             : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/logging.py
    reports             : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/reports.py
    python_path         : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/python_path.py
    unraisableexception : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/unraisableexception.py
    threadexception     : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/threadexception.py
    faulthandler        : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/faulthandler.py
    ddtrace             : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest/plugin.py
    ddtrace.pytest_bdd  : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest_bdd/plugin.py
    ddtrace.pytest_benchmark: /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest_benchmark/plugin.py
    xdist               : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/plugin.py
    xdist.looponfail    : /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/looponfail.py
    capturemanager      : <CaptureManager _method='fd' _global_capturing=<MultiCapture out=<FDCapture 1 oldfd=7 _state='suspended' tmpfile=<_io.TextIOWrapper name="<_io.FileIO name=8 mode='rb+' closefd=True>" mode='r+' encoding='utf-8'>> err=<FDCapture 2 oldfd=9 _state='suspended' tmpfile=<_io.TextIOWrapper name="<_io.FileIO name=10 mode='rb+' closefd=True>" mode='r+' encoding='utf-8'>> in_=<FDCapture 0 oldfd=5 _state='started' tmpfile=<_io.TextIOWrapper name='/dev/null' mode='r' encoding='utf-8'>> _state='suspended' _in_suspended=False> _capture_fixture=None>
    session             : <Session  exitstatus=<ExitCode.OK: 0> testsfailed=0 testscollected=0>
    lfplugin            : <_pytest.cacheprovider.LFPlugin object at 0x7f75edc3ea30>
    nfplugin            : <_pytest.cacheprovider.NFPlugin object at 0x7f75edc3ea00>
    legacypath-tmpdir   : <class '_pytest.legacypath.LegacyTmpdirPlugin'>
    terminalreporter    : <_pytest.terminal.TerminalReporter object at 0x7f75ed8c9970>
    logging-plugin      : <_pytest.logging.LoggingPlugin object at 0x7f75ed8c9f70>
    terminaldistreporter: <xdist.dsession.TerminalDistReporter object at 0x7f75ec0f3e80>
    dsession            : <xdist.dsession.DSession object at 0x7f75ec0f3340>
    funcmanage          : <_pytest.fixtures.FixtureManager object at 0x7f75ec0f3e50>
configfile: pyproject.toml
plugins: ddtrace-2.5.1, xdist-3.5.0
created: 4/4 workers
PLUGIN registered: <xdist.workermanage.WorkerController.RemoteHook object at 0x7f75ed40cb80>
PLUGIN registered: <xdist.workermanage.WorkerController.RemoteHook object at 0x7f75ed6d01c0>
PLUGIN registered: <xdist.workermanage.WorkerController.RemoteHook object at 0x7f75ed667c10>
PLUGIN registered: <xdist.workermanage.WorkerController.RemoteHook object at 0x7f75ed667670>
4 workers [30 items]

INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/main.py", line 272, in wrap_session
INTERNALERROR> E                 session.exitstatus = doit(config, session) or 0
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/main.py", line 326, in _main
INTERNALERROR> E                 config.hook.pytest_runtestloop(session=session)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR> E                 raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR> E                 teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/logging.py", line 796, in pytest_runtestloop
INTERNALERROR> E                 return (yield)  # Run all the tests.
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/remote.py", line [15], in pytest_runtestloop
INTERNALERROR> E                 self.run_one_test()
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/remote.py", line [17]4, in run_one_test
INTERNALERROR> E                 self.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.[18])/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 1[19], in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 181, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_result.py", line 99, in get_result
INTERNALERROR> E                 raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 87, in _multicall
INTERNALERROR> E                 next(wrapper_gen)  # first yield
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest/plugin.py", line 604, in pytest_runtest_protocol
INTERNALERROR> E                 test_module_span, module_is_package = _start_test_module_span(item)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest/plugin.py", line 279, in _start_test_module_span
INTERNALERROR> E                 test_module_span.set_tag_str(test.MODULE, item.config.hook.pytest_ddtrace_get_item_module_name(item=item))
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR> E                 raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest/plugin.py", line 835, in pytest_ddtrace_get_item_module_name
INTERNALERROR> E                 return pytest_package_item.module.__name__
INTERNALERROR> E             AttributeError: 'Package' object has no attribute 'module'
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/dsession.py:[20]: AssertionError
INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/main.py", line 272, in wrap_session
INTERNALERROR> E                 session.exitstatus = doit(config, session) or 0
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/main.py", line 326, in _main
INTERNALERROR> E                 config.hook.pytest_runtestloop(session=session)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR> E                 raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 1[21], in _multicall
INTERNALERROR> E                 teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/logging.py", line 796, in pytest_runtestloop
INTERNALERROR> E                 return (yield)  # Run all the tests.
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/remote.py", line 157, in pytest_runtestloop
INTERNALERROR> E                 self.run_one_test()
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/remote.py", line 174, in run_one_test
INTERNALERROR> E                 self.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 181, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_result.py", line 99, in get_result
INTERNALERROR> E                 raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 87, in _multicall
INTERNALERROR> E                 next(wrapper_gen)  # first yield
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest/plugin.py", line 604, in pytest_runtest_protocol
INTERNALERROR> E                 test_module_span, module_is_package = _start_test_module_span(item)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest/plugin.py", line 279, in _start_test_module_span
INTERNALERROR> E                 test_module_span.set_tag_str(test.MODULE, item.config.hook.pytest_ddtrace_get_item_module_name(item=item))
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR> E                 raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest/plugin.py", line 835, in pytest_ddtrace_get_item_module_name
INTERNALERROR> E                 return pytest_package_item.module.__name__
INTERNALERROR> E             AttributeError: 'Package' object has no attribute 'module'
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/dsession.py:200: AssertionError
INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/main.py", line 272, in wrap_session
INTERNALERROR> E                 session.exitstatus = doit(config, session) or 0
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/main.py", line 326, in _main
INTERNALERROR> E                 config.hook.pytest_runtestloop(session=session)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR> E                 raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR> E                 teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/logging.py", line 796, in pytest_runtestloop
INTERNALERROR> E                 return (yield)  # Run all the tests.
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/remote.py", line 157, in pytest_runtestloop
INTERNALERROR> E                 self.run_one_test()
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/remote.py", line 174, in run_one_test
INTERNALERROR> E                 self.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 181, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_result.py", line 99, in get_result
INTERNALERROR> E                 raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 87, in _multicall
INTERNALERROR> E                 next(wrapper_gen)  # first yield
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest/plugin.py", line 604, in pytest_runtest_protocol
INTERNALERROR> E                 test_module_span, module_is_package = _start_test_module_span(item)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest/plugin.py", line 279, in _start_test_module_span
INTERNALERROR> E                 test_module_span.set_tag_str(test.MODULE, item.config.hook.pytest_ddtrace_get_item_module_name(item=item))
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR> E                 raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest/plugin.py", line 835, in pytest_ddtrace_get_item_module_name
INTERNALERROR> E                 return pytest_package_item.module.__name__
INTERNALERROR> E             AttributeError: 'Package' object has no attribute 'module'
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/dsession.py:200: AssertionError
INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/main.py", line 272, in wrap_session
INTERNALERROR> E                 session.exitstatus = doit(config, session) or 0
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/main.py", line 326, in _main
INTERNALERROR> E                 config.hook.pytest_runtestloop(session=session)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR> E                 raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR> E                 teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/logging.py", line 796, in pytest_runtestloop
INTERNALERROR> E                 return (yield)  # Run all the tests.
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/remote.py", line 157, in pytest_runtestloop
INTERNALERROR> E                 self.run_one_test()
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/remote.py", line 174, in run_one_test
INTERNALERROR> E                 self.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 181, in _multicall
INTERNALERROR> E                 return outcome.get_result()
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_result.py", line 99, in get_result
INTERNALERROR> E                 raise exc.with_traceback(exc.__traceback__)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 87, in _multicall
INTERNALERROR> E                 next(wrapper_gen)  # first yield
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest/plugin.py", line 604, in pytest_runtest_protocol
INTERNALERROR> E                 test_module_span, module_is_package = _start_test_module_span(item)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest/plugin.py", line 279, in _start_test_module_span
INTERNALERROR> E                 test_module_span.set_tag_str(test.MODULE, item.config.hook.pytest_ddtrace_get_item_module_name(item=item))
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR> E                 raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest/plugin.py", line 835, in pytest_ddtrace_get_item_module_name
INTERNALERROR> E                 return pytest_package_item.module.__name__
INTERNALERROR> E             AttributeError: 'Package' object has no attribute 'module'
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> /opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/dsession.py:200: AssertionError
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/main.py", line 272, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/main.py", line 326, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/_pytest/logging.py", line 796, in pytest_runtestloop
INTERNALERROR>     return (yield)  # Run all the tests.
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/dsession.py", line 1[23], in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/xdist/dsession.py", line 1[37], in loop_once
INTERNALERROR>     raise RuntimeError("Unexpectedly no active workers available")
INTERNALERROR> RuntimeError: Unexpectedly no active workers available

What is the result that you expected?

My tests run without crashing

jhilimirejc commented 8 months ago

can confirm seeing the same issue in both 7.2.0 and 8.0.0

emmettbutler commented 8 months ago

Thanks for bringing this to our attention, @ravirahman. I've added a ticket about this to an internal backlog, and it will be included in future prioritization passes.

cc @romainkomorndatadog

romainkomorndatadog commented 8 months ago

I'm gonna reopen this since there's a comment about pytest 7.2.0, and also because I can't reproduce this naively with pytest 8.0.0 and Flask (which is my usual test "benchmark" repo).

I set up an environment with:

pip freeze ``` (ddtrace-2.5.1-py3.8.17) romain.komorn@COMP-CLXFH7L2J3 flask % pip freeze asgiref==3.7.2 attrs==23.2.0 blinker==1.7.0 bytecode==0.15.1 cattrs==23.2.3 click==8.1.7 coverage==7.4.1 ddsketch==2.0.4 ddtrace==2.5.1 Deprecated==1.2.14 envier==0.5.1 exceptiongroup==1.2.0 execnet==2.0.2 -e git+https://github.com/pallets/flask.git@c2f65dd1cfff0672b902fd5b30815f0b4137214c#egg=Flask importlib-metadata==6.11.0 iniconfig==2.0.0 itsdangerous==2.1.2 Jinja2==3.1.3 MarkupSafe==2.1.4 opentelemetry-api==1.22.0 packaging==23.2 pluggy==1.3.0 protobuf==4.25.2 pytest==8.0.0 pytest-xdist==3.5.0 python-dotenv==1.0.0 six==1.16.0 tomli==2.0.1 typing_extensions==4.9.0 Werkzeug==3.0.1 wrapt==1.16.0 xmltodict==0.13.0 zipp==3.17.0 ```

In both cases, I successfully ran tests with 

DD_SERVICE=flask DD_ENV=itr-on pytest --ddtrace -n auto

So whatever's going on isn't strictly related to pytest 8.0.0.

@ravirahman , are you only seeing this failure when you upgrade to 8.0.0 from 7.x?

@jhilimirejc , are you seeing the same traceback, give or take?

INTERNALERROR> E               File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/ddtrace/contrib/pytest/plugin.py", line 835, in pytest_ddtrace_get_item_module_name
INTERNALERROR> E                 return pytest_package_item.module.__name__
INTERNALERROR> E             AttributeError: 'Package' object has no attribute 'module'
romainkomorndatadog commented 8 months ago

@ravirahman , it'd be helpful if you could also try without using pytest-xdist, and if possible, how you're invoking pytest (ie the full CLI).

@jhilimirejc more details from you would be very welcome as well.

romainkomorndatadog commented 8 months ago

https://github.com/pytest-dev/pytest/issues/11137 does look like a potentially related change here, though if that were the case, it seems like I should definitely see a failure with 8.0.0, so I'll have to do some extra investigating.

There's a follow-up for me to have a default behavior for the module/suite/test naming hooks that doesn't cause exceptions (and if it does, that we just fail within CIVisibility and yield or return properly).

romainkomorndatadog commented 8 months ago

I can reproduce the same failure with Starlette ( git@github.com:encode/starlette.git ) at 4355e6bc288e14dea9597ad9c9265c0f9cad33c0, so @ravirahman , I don't think I need more from you at this point. Thanks for the report!

@jhilimirejc , if you could confirm whether you're seeing the same stacktrace with 7.2.0, that'd be great. If it's a different trace, could you open a different issue with details?

jhilimirejc commented 8 months ago

@romainkomorndatadog -We are not using pytest-xdist. However we had our version pinned at 1.20.2 and saw the same error, we attempted to upgrade and continue to see the same issue regardless of ddtrace or pytest versions

[tool.poetry.dependencies] python = "^3.10" PyGithub = "^1.51" GitPython = "^3.1.41" pyyaml = "^6.0.1" requests = "^2.31.0" azure-graphrbac = "0.61.0" braceexpand = "^0.1.7" junitparser = "^2.8.0" tabulate = "^0.9.0"

[tool.poetry.dev-dependencies] awscli = "^1.29.5" cryptography = ">=39.0.2" google-api-python-client = "^2.65.0" google-auth-oauthlib = "^0.7.0" google-auth-httplib2 = "^0.1.0" oauth2client = "^4.1.3" pyOpenSSL = "^23.0.0" pytest = "^7.2.0" python-ldap = "^3.3.1" typing_extensions = "^3.7.4" slackclient = "2.5.0" gherkin-official = "^24.1.0" Jinja2 = "3.0.3" /ddtrace and cattrs locked to specific versions due to requiring a typing_extensions upgrade to work correctly ddtrace = "1.20.2" cattrs = "22.2.0" pre-commit = "^2.2.0" sphinx = "^2.4.1" sphinx-gherkindoc = "^3.6.3" pytest-repeat = "^0.9.1" pdbpp = "^0.10.2" pyotp = "^2.4.1" tenacity = "^8.0.0"

(upgraded ddtrace, cattrs & attrs along with typing_extensions- and even tried pytest at 8.0.0 and still come across same error)

jhilimirejc commented 8 months ago

while our version of pytest is careted - i've pinned it to 7.20.0 and also see the same output:

INTERNALERROR> Traceback (most recent call last): INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/_pytest/main.py", line 272, in wrap_session INTERNALERROR> session.exitstatus = doit(config, session) or 0 INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/_pytest/main.py", line 326, in _main INTERNALERROR> config.hook.pytest_runtestloop(session=session) INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/pluggy/_hooks.py", line 501, in call INTERNALERROR> return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/pluggy/_manager.py", line 119, in _hookexec INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult) INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/pluggy/_callers.py", line 138, in _multicall INTERNALERROR> raise exception.with_traceback(exception.traceback) INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/pluggy/_callers.py", line 121, in _multicall INTERNALERROR> teardown.throw(exception) # type: ignore[union-attr] INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/_pytest/logging.py", line 796, in pytest_runtestloop INTERNALERROR> return (yield) # Run all the tests. INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/pluggy/_callers.py", line 102, in _multicall INTERNALERROR> res = hook_impl.function(args) INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/_pytest/main.py", line 351, in pytest_runtestloop INTERNALERROR> item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem) INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/pluggy/_hooks.py", line 501, in call INTERNALERROR> return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/pluggy/_manager.py", line 119, in _hookexec INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult) INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/pluggy/_callers.py", line 181, in _multicall INTERNALERROR> return outcome.get_result() INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/pluggy/_result.py", line 99, in get_result INTERNALERROR> raise exc.with_traceback(exc.traceback) INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/pluggy/_callers.py", line 87, in _multicall INTERNALERROR> next(wrapper_gen) # first yield INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/ddtrace/contrib/pytest/plugin.py", line 604, in pytest_runtest_protocol INTERNALERROR> test_module_span, module_is_package = _start_test_module_span(item) INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/ddtrace/contrib/pytest/plugin.py", line 279, in _start_test_module_span INTERNALERROR> test_module_span.set_tag_str(test.MODULE, item.config.hook.pytest_ddtrace_get_item_module_name(item=item)) INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/pluggy/_hooks.py", line 501, in call INTERNALERROR> return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/pluggy/_manager.py", line 119, in _hookexec INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult) INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/pluggy/_callers.py", line 138, in _multicall INTERNALERROR> raise exception.with_traceback(exception.traceback) INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/pluggy/_callers.py", line 102, in _multicall INTERNALERROR> res = hook_impl.function(args) INTERNALERROR> File "/usr/local/lib/python3.10/site-packages/ddtrace/contrib/pytest/plugin.py", line 835, in pytest_ddtrace_get_item_module_name INTERNALERROR> return pytest_package_item.module.name INTERNALERROR> AttributeError: 'Package' object has no attribute 'module'

iherasymenko commented 8 months ago

I put together a simple reproducer that can be found here: https://github.com/iherasymenko/ddtrace-8220-reproducer.

The key to trigger this error is to have a test file placed in a module.

romainkomorndatadog commented 7 months ago

As an update, I've got a WIP PR to support pytest 8.x in #8357 , but 8.x changes the order in which tests are processed, so I'm likely going to have to spend more time than I'd like refactoring tests so that they don't take order into account, so I'll probably have something mergeable on Wednesday (14 Feb).

@jhilimirejc , it would be very helpful if you could come up with a small repro case for Pytest 7.x, since neither 6.x nor 7.x fail on @iherasymenko 's reproducer. (edited to fix "Python 7.x" and double-negative "neither doesn't fail").

And @iherasymenko , thank you so much for that repro case. Definitely saved me some time.

romainkomorndatadog commented 7 months ago

For visibility here, I just wrote https://github.com/DataDog/dd-trace-py/pull/8357#issuecomment-1943345575 to say I need to take on a bit of a deeper rewrite of our CIVisibility service and pytest integration so that the changes made to support version 8.0 aren't just a set of temporary patches.

It'll be a slower turnaround but the changes are also needed for other work we have scheduled around telemetry and log collection.

tamird commented 7 months ago

Looks like this landed in main and in 2.6. Could someone cut a release for 2.6?

romainkomorndatadog commented 7 months ago

As I mentioned in https://github.com/DataDog/dd-trace-py/pull/8357#issuecomment-1948460944 , I ended up merging a short-term fix that addresses the reproducible crashes I could find.

For this repo, over the past 24 hours, we've had 235 successful test sessions where the pytest version is 8.0.0 or 8.0.1, and 25k passed tests. That's a good omen even if it doesn't entirely rule out edge cases I missed.

@tamird , let me see when we can get 2.6 out.

iherasymenko commented 7 months ago

Thanks for the fix @romainkomorndatadog. Our test suite is now all green with ddtrace 2.6.4 and pytest 8.0.1.

romainkomorndatadog commented 7 months ago

@tamird , I just released 2.6.4 and it passes the reproducer test.

(I wrote this hours ago and didn't hit the Comment button...)

@iherasymenko , thanks for confirming (and thanks again for the repro case :) ).

jhilimirejc commented 7 months ago

I didnt have a way to create a reproduction like @iherasymenko , sorry about that but can confirm that it's working now using 2.6.4 with pytest 8.0.1 also. thanks!