buildkite / test-collector-python

Python adapter for Buildkite Test Analytics
https://buildkite.com/test-analytics
MIT License
6 stars 5 forks source link

Crash when used with pytest-xdist #3

Closed matclayton closed 1 year ago

matclayton commented 2 years ago

Hey we're testing the plugin (0.1.1) with pytest 7.1.2 in the docker-compose#v3.9.0 on Python 3.10.3 and seeing the following crash

relevant parts of requirements.txt

pytest==7.1.2
pytest-asyncio==0.18.3
pytest-django==4.5.2
pytest-monkeytype==1.1.0
pytest-randomly==3.12.0
pytest-rerunfailures==10.2.0
pytest-sugar==0.9.5
pytest-timeout==2.1.0
pytest-xdist==2.5.0

Screenshot 2022-07-19 at 13 25 38

I'm not sure what other info would be helpful to debug this.

jimsynz commented 2 years ago

Hi @matclayton.

Huh. That's weird. It looks like it's trying to submit a payload that was never started - which is triggered by the pytest_runtestloop callback. Are you able to provide any more information about your test suite/setup?

matclayton commented 1 year ago

Happy to try and provide whatever details you need. We're using pytest with xdist and django, which I suspect might be the two suspect plugins causing this. Beyond that nothing is massively weird in the setup. I can try without xdist later on today, but trying without django won't result in any tests to run. I think all the key information is in this Issue, but I'll include the base printout as well below.

Screenshot 2022-09-07 at 11 32 48

pyproject.toml relevent parts


[tool.pytest.ini_options]
addopts = [
    "-n=auto",
    "--dist=loadscope",
    "--strict-markers",
    "--ds=mixcloud.settings.test",
    "--pdbcls=IPython.terminal.debugger:Pdb",
    "--durations=10",
    "--durations-min=0.5",
    "--asyncio-mode=auto",
]
xfail_strict = true```
matclayton commented 1 year ago

I've done some testing and If I remove pytest-xdist it works as expected, so the bug is likely in the interaction between the xdist package, which is a fairly common plugin for pytest and the collector I suspect. I also suspect anyone using analytics will care about performance and therefore running the tests in parallel is key to that as well, so I suspect the userbase will have a huge overlap :(

jimsynz commented 1 year ago

Thanks for that @matclayton - that gives me somewhere to start my investigation.

patrickoliveroblue commented 1 year ago

We're encountering the same issue as well in python 3.8.10.

Relevant section of requirements.txt file

pytest==7.1.3
pytest-forked==1.3.0
pytest-html==3.1.1
pytest-metadata==1.11.0
pytest-timeout==2.1.0
pytest-xdist==2.5.0
matclayton commented 1 year ago

Hey, sorry to nudge on this. are there any updates? given this is a paid service and the only open issue, its concerning that this might not be so well supported, despite it just launching.

jimsynz commented 1 year ago

Hi @matclayton - I've just opened #10 - any chance you can give that a go and double check that it works as expected?

Thanks!

matclayton commented 1 year ago

Amazing. Will give it a shot tomorrow asap