search

scientific-python / pytest-doctestplus

Pytest plugin providing advanced doctest features
Other
97 stars 38 forks source link

Doc tests not run with pytest 3.6 #24

Closed hamogu closed 6 years ago

hamogu commented 6 years ago

Since upgrading from pytest 3.5 to 3.6 the doctests are not run any longer. Compare, for example, https://travis-ci.org/Chandra-MARX/marxs/jobs/419227877#L2718 and https://travis-ci.org/Chandra-MARX/marxs/jobs/383848337#L3123 I don't know how that would be on pytest 3.7, because that already fails in the test collection stage as reported in #22. So, when testing locally, I'm freezing my conda environment to pytest 3.5.

drdavella commented 6 years ago

@hamogu thanks for the report. I can't reproduce this problem when running tests through pytest directly, so I wonder whether it's actually an issue with the Astropy test runner. I'll need to do some more investigation, though.

Just curious but does it actually work correctly for you locally when using pytest==3.5? The successful commit you showed me is from a few months ago and I'm not familiar with your project, so it's hard for me to isolate the exact cause.

hamogu commented 6 years ago

Yes it works locally with pytest 3.5. I set up a new conda environment, noticed that the documentation examples in /docs/*.rst where not run any longer. So, I downgraded pytest to 3.5.1 and run the tests for the same code and it worked. So, there must be something that changed between pytest 3.5 and 3.6. I checked the pytest changelog, but did not see anything that seemed related, so I decided to report is here. I see that inside my bigger project (and I obviously don't expect you to debug my project, I just wanted to give examples). I have not tried to construct a minimal working example with a fake, otherwise empty, project. That would take me quite some time and I hoped that you or @bsipocz would just look at this and go "Oh, that's what it is".

I don't see the tests in /docs/... run in other projects either, e.g. in astropy itself (https://travis-ci.org/astropy/astropy/jobs/419192478) or in photutils (https://travis-ci.org/astropy/photutils/jobs/418403363). In both cases, I'm pretty sure that is code inside the rst pages of the documentation.

drdavella commented 6 years ago

Thanks again for the data. I'm using this plugin for another project but I typically do not use the Astropy test runner for testing (instead I use just pytest) and do not see the issue there. I think I will probably proceed with a git bisect of Astropy since I strongly suspect this is not a problem with the plugin itself but rather with how the test runner executes the tests.

hamogu commented 6 years ago

In that case, it could also be in astropy-helpers. That's the downside of splitting out all these components - it's hard to know how exactly they interact. Thanks for looking into this!

bsipocz commented 6 years ago

@hamogu - Hmm, you're right the doctests from the rst files should have been picked up. Could you please open an issue in astropy core for it? It most probably will need a fix in an upstream repo elsewhere, but still is a valid issue for astropy. Sadly I most likely won't have much time in the coming days to investigate the reasons.

drdavella commented 6 years ago

@hamogu please see https://github.com/astropy/astropy/issues/7764. I suspect that the pytest version is a red herring, and that in a carefully controlled environment you would see that the only meaningful difference is the version of Astropy. The date of your last success on travis correlates well with the date of the offending PR in Astropy. However, I may be wrong so if you have additional input please let me know.

drdavella commented 6 years ago

@hamogu I'm going to close since this belongs to Astropy and has been documented in https://github.com/astropy/astropy/issues/7762. Thanks for bringing this to our attention!