search

DataDog / dd-trace-py

Datadog Python APM Client
https://ddtrace.readthedocs.io/
Other
547 stars 411 forks source link

2.2.0 raises a ValueError on windows #7625

Closed emmyoop closed 10 months ago

emmyoop commented 11 months ago

Summary of problem

We run tests with pytest on macos, ubuntu and windows.

Newly released version 2.2.0 fails when running on a windows runner. Issues resolved when I pin to 2.1.7.

Which version of dd-trace-py are you using?

2.2.0

Which version of pip are you using?

23.3.1

Which libraries and their versions are you using?

`pip freeze` agate==1.7.1 alabaster==0.7.13 appnope==0.1.3 asttokens==2.4.1 attrs==23.1.0 Babel==2.13.1 backcall==0.2.0 black==23.3.0 build==1.0.3 bump2version==1.0.1 bumpversion==0.6.0 bytecode==0.15.1 cachetools==5.3.2 cattrs==23.1.2 certifi==2023.7.22 cffi==1.16.0 cfgv==3.4.0 chardet==5.2.0 charset-normalizer==3.3.2 click==8.1.7 colorama==0.4.6 coverage==7.3.2 -e git+https://emmyoop@github.com/dbt-labs/dbt-core.git@c3860f8aa15437b1641b5cac7babfdefd3db9b4b#egg=dbt_core&subdirectory=core dbt-extractor==0.5.0 -e git+https://emmyoop@github.com/dbt-labs/dbt-core.git@c3860f8aa15437b1641b5cac7babfdefd3db9b4b#egg=dbt_postgres&subdirectory=plugins/postgres dbt-semantic-interfaces==0.4.1 -e git+https://emmyoop@github.com/dbt-labs/dbt-core.git@c3860f8aa15437b1641b5cac7babfdefd3db9b4b#egg=dbt_tests_adapter&subdirectory=tests/adapter ddsketch==2.0.4 ddtrace==2.1.6 decorator==5.1.1 Deprecated==1.2.14 distlib==0.3.7 docutils==0.20.1 envier==0.4.0 exceptiongroup==1.1.3 execnet==2.0.2 executing==2.0.1 filelock==3.13.1 flake8==6.1.0 flaky==3.7.0 freezegun==0.3.12 hypothesis==6.88.4 identify==2.5.31 idna==3.4 imagesize==1.4.1 importlib-metadata==6.8.0 importlib-resources==6.1.1 iniconfig==2.0.0 ipdb==0.13.13 ipython==8.12.3 isodate==0.6.1 jaraco.classes==3.3.0 jedi==0.19.1 Jinja2==3.1.2 jsonschema==4.19.2 jsonschema-specifications==2023.7.1 keyring==24.3.0 leather==0.3.4 Logbook==1.5.3 markdown-it-py==3.0.0 MarkupSafe==2.1.3 mashumaro==3.10 matplotlib-inline==0.1.6 mccabe==0.7.0 mdurl==0.1.2 minimal-snowplow-tracker==0.0.2 mocker==1.1.1 more-itertools==10.1.0 msgpack==1.0.7 mypy==1.4.1 mypy-extensions==1.0.0 networkx==3.1 nh3==0.2.14 nodeenv==1.8.0 opentelemetry-api==1.21.0 packaging==23.2 parsedatetime==2.6 parso==0.8.3 pathspec==0.11.2 pexpect==4.8.0 pickleshare==0.7.5 pip-tools==7.3.0 pkginfo==1.9.6 pkgutil_resolve_name==1.3.10 platformdirs==3.11.0 pluggy==1.3.0 pre-commit==3.5.0 prompt-toolkit==3.0.41 protobuf==4.25.0 psycopg2-binary==2.9.9 ptyprocess==0.7.0 pure-eval==0.2.2 pycodestyle==2.11.1 pycparser==2.21 pydantic==1.10.13 pyflakes==3.1.0 Pygments==2.16.1 pyproject-api==1.6.1 pyproject_hooks==1.0.0 pytest==7.4.3 pytest-cov==4.1.0 pytest-csv==3.0.0 pytest-dotenv==0.5.2 pytest-logbook==1.2.0 pytest-mock==3.12.0 pytest-split==0.8.1 pytest-xdist==3.4.0 python-dateutil==2.8.2 python-dotenv==1.0.0 python-slugify==8.0.1 pytimeparse==1.1.8 pytz==2023.3.post1 PyYAML==6.0.1 readme-renderer==42.0 referencing==0.30.2 requests==2.31.0 requests-toolbelt==1.0.0 rfc3986==2.0.0 rich==13.6.0 rpds-py==0.12.0 six==1.16.0 snowballstemmer==2.2.0 sortedcontainers==2.4.0 Sphinx==7.1.2 sphinxcontrib-applehelp==1.0.4 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.1 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 sqlparse==0.4.4 stack-data==0.6.3 text-unidecode==1.3 tomli==2.0.1 tox==4.11.3 traitlets==5.13.0 twine==4.0.2 types-colorama==0.4.15.12 types-docutils==0.20.0.3 types-freezegun==1.1.10 types-Jinja2==2.11.9 types-jsonschema==4.19.0.4 types-MarkupSafe==1.1.10 types-mock==5.1.0.2 types-protobuf==4.24.0.4 types-python-dateutil==2.8.19.14 types-pytz==2023.3.1.1 types-PyYAML==6.0.12.12 types-requests==2.30.0.0 types-setuptools==68.2.0.1 types-urllib3==1.26.25.14 typing_extensions==4.8.0 urllib3==1.26.18 virtualenv==20.24.6 wcwidth==0.2.10 wrapt==1.16.0 xmltodict==0.13.0 zipp==3.17.0

How can we reproduce your problem?

Run tests with ddtrace on a github windows runner.

What is the result that you get?

When using pytest, I get a Value error because the path changes.

logs ``` INTERNALERROR> Traceback (most recent call last): INTERNALERROR> File "D:\a\dbt-core\dbt-core\.tox\integration\lib\site-packages\_pytest\main.py", line 271, in wrap_session INTERNALERROR> session.exitstatus = doit(config, session) or 0 INTERNALERROR> File "D:\a\dbt-core\dbt-core\.tox\integration\lib\site-packages\_pytest\main.py", line 325, in _main INTERNALERROR> config.hook.pytest_runtestloop(session=session) INTERNALERROR> File "D:\a\dbt-core\dbt-core\.tox\integration\lib\site-packages\pluggy\_hooks.py", line [49](https://github.com/dbt-labs/dbt-core/actions/runs/6883250006/job/18723421574#step:8:50)3, in __call__ INTERNALERROR> return self._hookexec(self.name, self._hookimpls, kwargs, firstresult) INTERNALERROR> File "D:\a\dbt-core\dbt-core\.tox\integration\lib\site-packages\pluggy\_manager.py", line 115, in _hookexec INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult) INTERNALERROR> File "D:\a\dbt-core\dbt-core\.tox\integration\lib\site-packages\pluggy\_callers.py", line 130, in _multicall INTERNALERROR> teardown[0].send(outcome) INTERNALERROR> File "D:\a\dbt-core\dbt-core\.tox\integration\lib\site-packages\pluggy\_result.py", line 114, in get_result INTERNALERROR> raise exc.with_traceback(exc.__traceback__) INTERNALERROR> File "D:\a\dbt-core\dbt-core\.tox\integration\lib\site-packages\pluggy\_callers.py", line 77, in _multicall INTERNALERROR> res = hook_impl.function(*args) INTERNALERROR> File "D:\a\dbt-core\dbt-core\.tox\integration\lib\site-packages\_pytest\main.py", line 3[50](https://github.com/dbt-labs/dbt-core/actions/runs/6883250006/job/18723421574#step:8:51), in pytest_runtestloop INTERNALERROR> item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem) INTERNALERROR> File "D:\a\dbt-core\dbt-core\.tox\integration\lib\site-packages\pluggy\_hooks.py", line 493, in __call__ INTERNALERROR> return self._hookexec(self.name, self._hookimpls, kwargs, firstresult) INTERNALERROR> File "D:\a\dbt-core\dbt-core\.tox\integration\lib\site-packages\pluggy\_manager.py", line 115, in _hookexec INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult) INTERNALERROR> File "D:\a\dbt-core\dbt-core\.tox\integration\lib\site-packages\pluggy\_callers.py", line 1[52](https://github.com/dbt-labs/dbt-core/actions/runs/6883250006/job/18723421574#step:8:53), in _multicall INTERNALERROR> return outcome.get_result() INTERNALERROR> File "D:\a\dbt-core\dbt-core\.tox\integration\lib\site-packages\pluggy\_result.py", line 114, in get_result INTERNALERROR> raise exc.with_traceback(exc.__traceback__) INTERNALERROR> File "D:\a\dbt-core\dbt-core\.tox\integration\lib\site-packages\pluggy\_callers.py", line [62](https://github.com/dbt-labs/dbt-core/actions/runs/6883250006/job/18723421574#step:8:63), in _multicall INTERNALERROR> next(wrapper_gen) # first yield INTERNALERROR> File "D:\a\dbt-core\dbt-core\.tox\integration\lib\site-packages\ddtrace\contrib\pytest\plugin.py", line [64](https://github.com/dbt-labs/dbt-core/actions/runs/6883250006/job/18723421574#step:8:65)5, in pytest_runtest_protocol INTERNALERROR> _add_start_end_source_file_path_data_to_span(span, test_method_object, test_name) INTERNALERROR> File "D:\a\dbt-core\dbt-core\.tox\integration\lib\site-packages\ddtrace\internal\ci_visibility\utils.py", line 40, in _add_start_end_source_file_path_data_to_span INTERNALERROR> source_file_path = get_source_file_path_for_test_method(test_method_object) INTERNALERROR> File "D:\a\dbt-core\dbt-core\.tox\integration\lib\site-packages\ddtrace\internal\ci_visibility\utils.py", line 15, in get_source_file_path_for_test_method INTERNALERROR> source_file_path = os.path.relpath(inspect.getfile(test_method_object)) INTERNALERROR> File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\ntpath.py", line [70](https://github.com/dbt-labs/dbt-core/actions/runs/6883250006/job/18723421574#step:8:71)3, in relpath INTERNALERROR> raise ValueError("path is on mount %r, start on mount %r" % ( INTERNALERROR> ValueError: path is on mount 'D:', start on mount 'C:' ```

example run

What is the result that you expected?

For all my tests to run.

Extra info

I think the problem is with the use of relpath here because the path and start are on different drives, per the docs

romainkomorndatadog commented 11 months ago

@emmyoop , thanks for the report details. We're going to look at this today. Just to confirm, looking at the exception in your example run, do I have this correct?

When you start the command from this log entry integration: commands[0]> .tox\integration\Scripts\python.exe -m pytest --cov=core --ddtrace tests/functional, does that happen on D: or C: ? Based on what I'm seeing, I want to say that's on D: but if that were the case, the error you're getting seems unexpected?

emmyoop commented 11 months ago

That all looks correct to me as well. 🤔 I'll take a closer look later today too to see if I can determine what's happening.

For what it's worth, we're not doing any path manipulation with GitHub and we use the GitHub managed actions to checkout the repo and install python.

romainkomorndatadog commented 11 months ago

I think we have a good workaround in progress in https://github.com/DataDog/dd-trace-py/pull/7627/ .

romainkomorndatadog commented 11 months ago

@emmyoop , if you have a chance, would you be able to run a test with the following? 

pip install git+https://github.com/DataDog/dd-trace-py.git@54419ef6a3523776d803d9f4cba2334dcceb9fbb

That should resolve your issue since your pytest's rootdir is D:\a\dbt-core\dbt-core (according to https://github.com/dbt-labs/dbt-core/actions/runs/6883250006/job/18723421574#step:8:34) .

That should prevent the INTERNALERROR> ValueError: path is on mount 'D:', start on mount 'C:' error.

emmyoop commented 11 months ago

I don't actually have a windows machine. I modified our dependencies in this PR and the windows tests in GitHub ran successfully! The fix has resolved our issues. Thanks so much!

We're pinned to 2.1.7 so no rush but what's the general cadence for releasing bug fixes?

romainkomorndatadog commented 11 months ago

@emmyoop , I think we should get it into 2.3.0 which should be out today or tomorrow. Looks like we just missed the cut on the next patch releases for 2.2, though, so that may have to wait another week.

ericlaz commented 11 months ago

@emmyoop, as an update, 2.3.0 is now out which should fix the ValueError issue. Hope this helps!

romainkomorndatadog commented 11 months ago

https://github.com/DataDog/dd-trace-py/pull/7680 merged into our 2.2 release branch, so it should also make it out in 2.2.1 which I think we're hoping to release today.

romainkomorndatadog commented 11 months ago

2.2.1 was published a little while ago. @emmyoop , I'd rather wait for your thumbs up before closing this out, if you don't mind.

emmyoop commented 10 months ago

Sorry for the late reply. I've been off for the last several days. I can confirm this fixed it for us in both 2.3.0 and 2.2.1. Thanks so much for the quick turn around!

romainkomorndatadog commented 10 months ago

Great! I'll close this now then. :) Thanks to @ericlaz for the quick fix!