Version 0.6.1 leads to INTERNALERROR on Azure pipelines

[bwinkel]

Answer

https://github.com/astropy/pytest-doctestplus/issues/111#issuecomment-1323417858

Original post

I encountered a strange bug, which seems to be related with doctestplus. I use it (as part of the Astropy package template machinery) for my pycraf package. The CI/CD is using Azure pipelines.

With the new 0.6.1 version I get a strange error in my tests:

INTERNALERROR> ModuleNotFoundError: No module named 'importing_pycraf'

(Full log file on Azure pipelines)

The noteworthy thing here is, that there is no module "importing_pycraf" in my package (only "pycraf" itself) and I don't get the error on my local machine (with the same package versions installed).

It was quite difficult to trace this back to doctestplus, but the error goes away, if I pin the version to 0.6.0; see here.

Any help would be appreciated. And thanks a lot for providing this nice little plugin :smile:

[bsipocz]

@bwinkel - I see that you still have the version pinned in your CI, could you maybe double check whether you still need that pin, or somehow this cryptic issue has been resolved over the past years?

[bwinkel]

Thanks for pinging me. Unfortunately, there are still some issues, although the error message has slightly change. Note that the issue only persists on older python versions on linux (currently the oldest supported version in my project is 3.8). So I guess this can be closed.

[pllim]

Python 3.8 isn't that old though. 🤔

[bsipocz]

I run into something similar with photutils, but for all python versions, depending on where the contest file was (so it wasn't doctestplus related)

[pllim]

Closing then. Thanks for the updates!

[bsipocz]

I'm not saying it's the same, but similar. Otherwise I don't have a working idea for the cause of the issue.

[bwinkel]

I run into something similar with photutils, but for all python versions, depending on where the contest file was (so it wasn't doctestplus related)

Would you be able to elaborate? I'm now getting similar errors too on all python versions and archs (but only on Azure). What is even more strange is that I have another package, where it just runs fine 🤔

For a moment, I thought, it might be related to the structure of my docs folder. I had a folder for each sub-package with index.rst in it. But even if I rename them to e.g. atm.rst and put them all in the top-level docs folder, I get the following:

============================= test session starts ==============================
platform linux -- Python 3.9.13, pytest-7.2.0, pluggy-1.0.0

Running tests with Astropy version 5.1.1.
Running tests in docs.

Date: 2022-11-21T09:24:18

Platform: Linux-5.15.0-1022-azure-x86_64-with-glibc2.31

Executable: /usr/share/miniconda/envs/pycraf-env/bin/python

Full Python Version: 
3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:58:50) 
[GCC 10.3.0]

encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15

Package versions: 
Numpy: 1.20.3
Scipy: 1.9.3
Matplotlib: 3.3.2
h5py: 3.7.0
Pandas: not available

Using Astropy options: remote_data: any.

rootdir: /home/vsts/work/1/s, configfile: setup.cfg
plugins: filter-subpackage-0.1.1, hypothesis-6.58.0, openfiles-0.5.0, azurepipelines-1.0.4, nunit-1.0.3, arraydiff-0.3, astropy-0.10.0, doctestplus-0.12.1, remotedata-0.3.3, mock-3.10.0, astropy-header-0.1.2, cov-4.0.0
collected 8 items
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/_pytest/python.py", line 618, in _importtestmodule
INTERNALERROR>     mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/_pytest/pathlib.py", line 533, in import_path
INTERNALERROR>     importlib.import_module(module_name)
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/importlib/__init__.py", line 127, in import_module
INTERNALERROR>     return _bootstrap._gcd_import(name[level:], package, level)
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
INTERNALERROR> ModuleNotFoundError: No module named 'atm'

INTERNALERROR> 
INTERNALERROR> The above exception was the direct cause of the following exception:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/_pytest/main.py", line 270, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/_pytest/main.py", line 334, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/_pytest/main.py", line 667, in perform_collect
INTERNALERROR>     hook.pytest_collection_modifyitems(
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/pytest_nunit/plugin.py", line 352, in pytest_collection_modifyitems
INTERNALERROR>     if item.parent and hasattr(item.parent, "obj") and item.parent.obj:
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/_pytest/python.py", line 311, in obj
INTERNALERROR>     self._obj = obj = self._getobj()
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/_pytest/python.py", line 529, in _getobj
INTERNALERROR>     return self._importtestmodule()
INTERNALERROR>   File "/usr/share/miniconda/envs/pycraf-env/lib/python3.9/site-packages/_pytest/python.py", line 643, in _importtestmodule
INTERNALERROR>     raise self.CollectError(
INTERNALERROR> _pytest.nodes.Collector.CollectError: ImportError while importing test module '/home/vsts/work/1/s/docs/atm.rst'.
INTERNALERROR> Hint: make sure your test modules/packages have valid Python names.
INTERNALERROR> Traceback:
INTERNALERROR> /usr/share/miniconda/envs/pycraf-env/lib/python3.9/importlib/__init__.py:127: in import_module
INTERNALERROR>     return _bootstrap._gcd_import(name[level:], package, level)
INTERNALERROR> E   ModuleNotFoundError: No module named 'atm'
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
No Result Found to Publish '/home/vsts/work/1/s/docs/test-output.xml'.
##[warning]Coverage XML was not created, skipping upload.

============================ no tests ran in 0.14s =============================
##[error]Bash exited with code '3'.

Note that my conftest.py is in the package folder and is basically the same as the one in the photutils package.

[bsipocz]

@bwinkel - I can't recall the details anymore (it was in https://github.com/astropy/photutils/pull/1410, so the action logs may still be around), but basically I was experimenting with placing the conftest.py file at different locations to ensure it's picked up both for the packaged version as well as for CI/tox.

[bwinkel]

OK, thanks a lot for your help @bsipocz. I'll keep trying. The odd thing is that it works with an older version of doctestplus. Really strange.

Should I be successful, I'll let you know.

[bsipocz]

Thanks. I still feel it's not really doctestplus related, but nevertheless, any clues unearthed would be appreciated (I really had to drop the ball on it as it was such a sidetracked rabbit hole at the time).

[pllim]

What version of pip is being used in your Azure pipeline? There are a bunch of similar questions on the internet and the prescribed solutions (if any) were all a little different. 🤯

[bsipocz]

@pllim - this is not limited to azure any more, I've run into it on GHA.

[pllim]

Azure and Actions both owned by Microsoft now, so there is a small chance it is the same problem... 😬

[bwinkel]

I'm using a relatively new pip. That was also a source of a lot of confusion. At some point they started to use an isolated build environment - which did not work well in my handcrafted conda environment on Azure. I solved it by using the following:

python -m pip install --no-build-isolation --no-deps .

To be precise, I do an editable install with the -e option, such that I can run pytest from the project directory. I also suspected this to be a possible cause, but as I said, it's working for another package of mine without problems.

[bsipocz]

Azure and Actions both owned by Microsoft now, so there is a small chance it is the same problem...

I still don't think that's a root of the issue. GHA uses the latest pip: https://github.com/astropy/photutils/actions/runs/3029597703/jobs/4901556371

[pllim]

Do you even use https://pypi.org/project/atm/ somewhere in your dep tree? If so, someone else on the internet suggested inserting that package into your requirements file. 🤷

[bwinkel]

Oh, there is a misunderstanding. atm is a sub-package in pycraf. I guess, pytest is importing it either for the tests or for the API docs.

[bwinkel]

If this would at least happen on my desktop machine, then I'd fire up pdb and see, what is happening exactly. But Azure is so incredibly hard to debug 😢

[pllim]

Ah, does this atm module have Cython or C-extension? If yes, sometimes the import error is just a symptom of build error.

[bwinkel]

Yes, it does. However, some of the other sub-packages don't. And the error occurs for all of them. (Of course it could be that it's because the package contains cython extensions in general.)

Before I run the tests on the doc-folder, I run the tests in the package folder - which also includes doc-strings, which are successfully tested. Weird, isn't it???

[bsipocz]

hmm, the extensions could be at fault, indeed. Locally for photutils:

$ ~/munka/devel/photutils [conftest_to_have_header_for_tox L|…1⚑ 17] $ git clean -dfx -e .idea -e notes -e benchmarks -e .github
Removing .tox/
Removing photutils.egg-info/
Removing photutils/_compiler.c
Removing photutils/version.py
$ ~/munka/devel/photutils [conftest_to_have_header_for_tox L|…1⚑ 17] $ tox -e test
.package create: /Users/bsipocz/munka/devel/photutils/.tox/.package
.package installdeps: setuptools, setuptools_scm, wheel, cython>=0.29.22, oldest-supported-numpy, extension-helpers
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
/Users/bsipocz/munka/devel/photutils/.tox/.package/lib/python3.10/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
  warnings.warn(msg, warning_class)
/Users/bsipocz/munka/devel/photutils/.tox/.package/lib/python3.10/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
  warnings.warn(msg, warning_class)
no previously-included directories found matching 'build'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/api'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.o' found anywhere in distribution
test create: /Users/bsipocz/munka/devel/photutils/.tox/test
test inst: /Users/bsipocz/munka/devel/photutils/.tox/.tmp/package/1/photutils-1.5.1.dev35+g26f64e33.tar.gz
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test installed: astropy==5.1.1,attrs==22.1.0,coverage==6.5.0,exceptiongroup==1.0.4,hypothesis==6.58.0,iniconfig==1.1.1,numpy==1.23.5,packaging==21.3,photutils==1.5.1.dev35+g26f64e33,pluggy==1.0.0,psutil==5.9.4,pyerfa==2.0.0.1,pyparsing==3.0.9,pytest==7.2.0,pytest-arraydiff==0.5.0,pytest-astropy==0.10.0,pytest-astropy-header==0.2.2,pytest-cov==4.0.0,pytest-doctestplus==0.12.1,pytest-filter-subpackage==0.1.1,pytest-mock==3.10.0,pytest-openfiles==0.5.0,pytest-remotedata==0.3.3,PyYAML==6.0,sortedcontainers==2.4.0,tomli==2.0.1
test run-test-pre: PYTHONHASHSEED='654182927'
test run-test: commands[0] | pip freeze
astropy==5.1.1
attrs==22.1.0
coverage==6.5.0
exceptiongroup==1.0.4
hypothesis==6.58.0
iniconfig==1.1.1
numpy==1.23.5
packaging==21.3
photutils @ file:///Users/bsipocz/munka/devel/photutils/.tox/.tmp/package/1/photutils-1.5.1.dev35%2Bg26f64e33.tar.gz
pluggy==1.0.0
psutil==5.9.4
pyerfa==2.0.0.1
pyparsing==3.0.9
pytest==7.2.0
pytest-arraydiff==0.5.0
pytest-astropy==0.10.0
pytest-astropy-header==0.2.2
pytest-cov==4.0.0
pytest-doctestplus==0.12.1
pytest-filter-subpackage==0.1.1
pytest-mock==3.10.0
pytest-openfiles==0.5.0
pytest-remotedata==0.3.3
PyYAML==6.0
sortedcontainers==2.4.0
tomli==2.0.1
test run-test: commands[1] | pytest --pyargs photutils /Users/bsipocz/munka/devel/photutils/docs
<frozen importlib._bootstrap>:241: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 96 from PyObject
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/bsipocz/munka/devel/photutils/.tox/test/lib/python3.10/site-packages/_pytest/main.py", line 266, in wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File "/Users/bsipocz/munka/devel/photutils/.tox/test/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1037, in _do_configure
INTERNALERROR>     self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File "/Users/bsipocz/munka/devel/photutils/.tox/test/lib/python3.10/site-packages/pluggy/_hooks.py", line 277, in call_historic
INTERNALERROR>     res = self._hookexec(self.name, self.get_hookimpls(), kwargs, False)
INTERNALERROR>   File "/Users/bsipocz/munka/devel/photutils/.tox/test/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/bsipocz/munka/devel/photutils/.tox/test/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/bsipocz/munka/devel/photutils/.tox/test/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/Users/bsipocz/munka/devel/photutils/.tox/test/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/bsipocz/munka/devel/photutils/conftest.py", line 31, in pytest_configure
INTERNALERROR>     from photutils import __version__
INTERNALERROR>   File "/Users/bsipocz/munka/devel/photutils/photutils/__init__.py", line 15, in <module>
INTERNALERROR>     from .aperture import *  # noqa
INTERNALERROR>   File "/Users/bsipocz/munka/devel/photutils/photutils/aperture/__init__.py", line 7, in <module>
INTERNALERROR>     from .circle import *  # noqa
INTERNALERROR>   File "/Users/bsipocz/munka/devel/photutils/photutils/aperture/circle.py", line 15, in <module>
INTERNALERROR>     from ..geometry import circular_overlap_grid
INTERNALERROR>   File "/Users/bsipocz/munka/devel/photutils/photutils/geometry/__init__.py", line 6, in <module>
INTERNALERROR>     from .circular_overlap import *  # noqa
INTERNALERROR> ModuleNotFoundError: No module named 'photutils.geometry.circular_overlap'
ERROR: InvocationError for command /Users/bsipocz/munka/devel/photutils/.tox/test/bin/pytest --pyargs photutils /Users/bsipocz/munka/devel/photutils/docs (exited with code 3)
________________________________________________________________________________ summary ________________________________________________________________________________
ERROR:   test: commands failed
$ ~/munka/devel/photutils [conftest_to_have_header_for_tox L|…1⚑ 17] $ rm conftest.py 
$ ~/munka/devel/photutils [conftest_to_have_header_for_tox L|✚ 1…1⚑ 17] $ git clean -dfx -e .idea -e notes -e benchmarks -e .github
Removing .tmp/
Removing .tox/
Removing __pycache__/
Removing photutils.egg-info/
Removing photutils/__pycache__/
Removing photutils/_compiler.c
Removing photutils/aperture/__pycache__/
Removing photutils/geometry/__pycache__/
Removing photutils/utils/__pycache__/
Removing photutils/version.py
$ ~/munka/devel/photutils [conftest_to_have_header_for_tox L|✚ 1…1⚑ 17] $ tox -e test
.package create: /Users/bsipocz/munka/devel/photutils/.tox/.package
.package installdeps: setuptools, setuptools_scm, wheel, cython>=0.29.22, oldest-supported-numpy, extension-helpers
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
/Users/bsipocz/munka/devel/photutils/.tox/.package/lib/python3.10/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
  warnings.warn(msg, warning_class)
/Users/bsipocz/munka/devel/photutils/.tox/.package/lib/python3.10/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
  warnings.warn(msg, warning_class)
no previously-included directories found matching 'build'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/api'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.o' found anywhere in distribution
test create: /Users/bsipocz/munka/devel/photutils/.tox/test
test inst: /Users/bsipocz/munka/devel/photutils/.tox/.tmp/package/1/photutils-1.5.1.dev35+g26f64e33.d20221121.tar.gz
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test installed: astropy==5.1.1,attrs==22.1.0,coverage==6.5.0,exceptiongroup==1.0.4,hypothesis==6.58.0,iniconfig==1.1.1,numpy==1.23.5,packaging==21.3,photutils==1.5.1.dev35+g26f64e33.d20221121,pluggy==1.0.0,psutil==5.9.4,pyerfa==2.0.0.1,pyparsing==3.0.9,pytest==7.2.0,pytest-arraydiff==0.5.0,pytest-astropy==0.10.0,pytest-astropy-header==0.2.2,pytest-cov==4.0.0,pytest-doctestplus==0.12.1,pytest-filter-subpackage==0.1.1,pytest-mock==3.10.0,pytest-openfiles==0.5.0,pytest-remotedata==0.3.3,PyYAML==6.0,sortedcontainers==2.4.0,tomli==2.0.1
test run-test-pre: PYTHONHASHSEED='1044436757'
test run-test: commands[0] | pip freeze
astropy==5.1.1
attrs==22.1.0
coverage==6.5.0
exceptiongroup==1.0.4
hypothesis==6.58.0
iniconfig==1.1.1
numpy==1.23.5
packaging==21.3
photutils @ file:///Users/bsipocz/munka/devel/photutils/.tox/.tmp/package/1/photutils-1.5.1.dev35%2Bg26f64e33.d20221121.tar.gz
pluggy==1.0.0
psutil==5.9.4
pyerfa==2.0.0.1
pyparsing==3.0.9
pytest==7.2.0
pytest-arraydiff==0.5.0
pytest-astropy==0.10.0
pytest-astropy-header==0.2.2
pytest-cov==4.0.0
pytest-doctestplus==0.12.1
pytest-filter-subpackage==0.1.1
pytest-mock==3.10.0
pytest-openfiles==0.5.0
pytest-remotedata==0.3.3
PyYAML==6.0
sortedcontainers==2.4.0
tomli==2.0.1
test run-test: commands[1] | pytest --pyargs photutils /Users/bsipocz/munka/devel/photutils/docs
========================================================================== test session starts ==========================================================================
platform darwin -- Python 3.10.0, pytest-7.2.0, pluggy-1.0.0
cachedir: .tox/test/.pytest_cache
Running tests in photutils docs.

Date: 2022-11-21T14:09:44

Platform: macOS-10.16-x86_64-i386-64bit

Executable: /Users/bsipocz/munka/devel/photutils/.tox/test/bin/python

Full Python Version: 
3.10.0 (default, Nov  1 2021, 16:57:20) [Clang 10.0.0 (clang-1000.11.45.5)]

encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15

Package versions: 
Numpy: 1.23.5
Scipy: not available
Matplotlib: not available
h5py: not available
Pandas: not available

Using Astropy options: remote_data: none.

rootdir: /Users/bsipocz/munka/devel/photutils, configfile: setup.cfg
plugins: remotedata-0.3.3, hypothesis-6.58.0, mock-3.10.0, astropy-header-0.2.2, doctestplus-0.12.1, astropy-0.10.0, filter-subpackage-0.1.1, cov-4.0.0, openfiles-0.5.0, arraydiff-0.5.0
collected 1480 items                                                                                                                                                    

../../.tox/test/lib/python3.10/site-packages/photutils/aperture/bounding_box.py ..                                                                                [  0%]
../../.tox/test/lib/python3.10/site-packages/photutils/aperture/circle.py ....                                                                                    [  0%]
../../.tox/test/lib/python3.10/site-packages/photutils/aperture/ellipse.py ....                                                                                   [  0%]
../../.tox/test/lib/python3.10/site-packages/photutils/aperture/rectangle.py ....                                                                                 [  0%]
../../.tox/test/lib/python3.10/site-packages/photutils/aperture/stats.py .                                                                                        [  1%]
../../.tox/test/lib/python3.10/site-packages/photutils/aperture/tests/test_bounding_box.py ..........sss..                                                        [  2%]
../../.tox/test/lib/python3.10/site-packages/photutils/aperture/tests/test_circle.py ..ss......ss...................                                              [  4%]
../../.tox/test/lib/python3.10/site-packages/photutils/aperture/tests/test_ellipse.py ..........................                                                  [  5%]
../../.tox/test/lib/python3.10/site-packages/photutils/aperture/tests/test_mask.py .......................                                                        [  7%]
../../.tox/test/lib/python3.10/site-packages/photutils/aperture/tests/test_photometry.py ............ssssss...........s.............ss....                        [ 10%]
../../.tox/test/lib/python3.10/site-packages/photutils/aperture/tests/test_rectangle.py ..........................                                                [ 12%]
../../.tox/test/lib/python3.10/site-packages/photutils/aperture/tests/test_stats.py ....................                                                          [ 13%]
../../.tox/test/lib/python3.10/site-packages/photutils/background/core.py .........                                                                               [ 14%]
../../.tox/test/lib/python3.10/site-packages/photutils/background/tests/test_background_2d.py ssssssssssssssssssssssssssssssssssssssss                            [ 17%]
../../.tox/test/lib/python3.10/site-packages/photutils/background/tests/test_core.py ............................................................................ [ 22%]
.                                                                                                                                                                 [ 22%]
../../.tox/test/lib/python3.10/site-packages/photutils/centroids/tests/test_core.py ssssssssssssssssssssssssss                                                    [ 24%]
../../.tox/test/lib/python3.10/site-packages/photutils/centroids/tests/test_gaussian.py sssssssssssss.                                                            [ 25%]
../../.tox/test/lib/python3.10/site-packages/photutils/datasets/load.py .                                                                                         [ 25%]
../../.tox/test/lib/python3.10/site-packages/photutils/datasets/make.py ....                                                                                      [ 25%]
../../.tox/test/lib/python3.10/site-packages/photutils/datasets/tests/test_load.py ..s                                                                            [ 25%]
../../.tox/test/lib/python3.10/site-packages/photutils/datasets/tests/test_make.py .......s........ss                                                             [ 26%]
../../.tox/test/lib/python3.10/site-packages/photutils/detection/tests/test_daofinder.py sssssssssssssssssssss                                                    [ 28%]
../../.tox/test/lib/python3.10/site-packages/photutils/detection/tests/test_irafstarfinder.py sssssssssssssssssss                                                 [ 29%]
../../.tox/test/lib/python3.10/site-packages/photutils/detection/tests/test_peakfinder.py sssssssssssssss                                                         [ 30%]
../../.tox/test/lib/python3.10/site-packages/photutils/detection/tests/test_starfinder.py sssssss                                                                 [ 31%]
../../.tox/test/lib/python3.10/site-packages/photutils/geometry/tests/test_circular_overlap_grid.py ......................................................        [ 34%]
../../.tox/test/lib/python3.10/site-packages/photutils/geometry/tests/test_elliptical_overlap_grid.py ........................................................... [ 38%]
................................................................................................................................................................. [ 49%]
..............................................................................................^C

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/Users/bsipocz/munka/devel/photutils/.tox/test/lib/python3.10/site-packages/photutils/geometry/tests/test_elliptical_overlap_grid.py:33: KeyboardInterrupt
(to show a full traceback on KeyboardInterrupt use --full-trace)
=================================================================== 666 passed, 161 skipped in 8.51s ====================================================================
ERROR: got KeyboardInterrupt signal
________________________________________________________________________________ summary ________________________________________________________________________________
ERROR:   test: keyboardinterrupt

[pllim]

Before I run the tests on the doc-folder, I run the tests in the package folder - which also includes doc-strings, which are successfully tested. Weird, isn't it???

I am just wildly guessing now but perhaps the API docstring that lives in the code folder can somehow find that submodule or does not need it at all, but doctest in a separate docs directory failed for some reason.

I mean, if you move that offending doctest from "docs" to some API docstring, does it then pass or fail on Azure?

[bsipocz]

Azure is a red herring, I see this locally, too, and I see this when doctestplus is not being involved, too (doctest-rst is removed from setup.cfg as well as doctestplus uninstalled)

[bwinkel]

<frozen importlib._bootstrap>:241: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 96 from PyObject

You have another problem in your case, @bsipocz. You probably use a "wrong" numpy version somewhere. But I made sure, that in my case this is not the issue. Otherwise, it may have been the culprit, as it could cause a crash on Azure.

[bwinkel]

Also, in your case the traceback suggests that it happens when you first import the package (for the version number in conftest.py). But in my case, the traceback doesn't contain a single import pycraf (or sub-package import)...

[bsipocz]

that's (==numpy) another red herring, the point is that the extensions are not built if/when the conftest.py is in the wrong directory. Now, the issue is that I don't yet see why it's "wrong"

[bwinkel]

But in my case, the extensions are built. I do this in a previous step (and successfully, as the normal tests run fine). Keep in mind that I'm not using tox. There is no extra (tox or pip) build isolation in my case, just the fact that I'm using pristine Azure VMs for each python version and arch.

[bwinkel]

https://dev.azure.com/bwinkel78/Benjamin-Winkel-Projects/_build/results?buildId=560&view=logs&j=6fabfd88-d052-5ecb-124c-95e1da1a9a76

(I'll remove the link later, as Azure deletes the log after a while)

[pllim]

I don't understand why you are running pytest twice. The second command should have run both the actual tests and also doctests. What if you comment out the first python -m pytest -rsx --remote-data=any --ignore-glob="*/setup_package.py" pycraf ? (Not sure if that will fix your error but who knows...)

I also don't grok all the Azure stuff that is going on in the log. I see your pyx got build and then shoved into a /tmp dir somewhere but then I lost track of when it is supposed to be put into your site-packages.

Even before astropy adopted tox, we ditched conda a long time before that in the test infrastructure. So I don't know what conda/mamba is doing there. Is there a chance there are multiple copies of pip or whatever in your env, and the system got confused?

Sorry, I think I am at my wits end here. 😞 Hopefully @bsipocz or another dev that is more proficient in Azure + conda have more ideas.

[pllim]

Also I don't quite remember what --doctest-ignore-import-errors was supposed to do, but is there a chance it is masking out some real problem for you?

[bwinkel]

I don't understand why you are running pytest twice.

The reason for this is that I normally don't test the docs folder on all archs, but only in one of the VMs (to save some computing time).

Also I don't quite remember what --doctest-ignore-import-errors was supposed to do, but is there a chance it is masking out some real problem for you?

Let me try without this...

[bwinkel]

Also I don't quite remember what --doctest-ignore-import-errors was supposed to do, but is there a chance it is masking out some real problem for you?

Let me try without this...

Doesn't make a difference.

[bsipocz]

I might have lost this point somewhere in the comment thread above, but have you tried this without doctestplus and succeeded? That is both making sure doctestplus is not used, and preferably not even installed.

[bwinkel]

Well the tests won't succeed, because I use FLOAT_CMP a lot. But using an old version of doctestplus (0.6 or so) works...

[bsipocz]

Sure, the test would fail, but that only once the build and collection and running is successful. Do you get through all that without doctestplus?

[bwinkel]

Good idea. I'll try tomorrow.

[bwinkel]

OK, so I expected, without doctestplus the tests run just "fine" (of course they all fail because of the missing directives, but the rst-files are processed normally).

[saimn]

It seems there is an issue or conflict with pytest-azurepipelines / pytest-nunit. I can reproduce the error only if I install those 2 packages and run the unit tests before the doctestplus run.

[bwinkel]

I'll try that immediately. In fact, in the other package (which works) I don't use pytest-azurepipelines. Slipped my eye.

Let's cross fingers.

[bwinkel]

I think that did the trick! There are some other errors now but that's unrelated to doctestplus.

Thank you so much for your help, all!

[pllim]

Wow, I am so glad Simon chimed in. I would never have thought of that. Thanks!

[bwinkel]

Everything works now like a charm. You people are amazing!

@saimn Do you have an idea whether the bug is with doctestplus or pytest-azurepipelines? Should one try to fix it?

[saimn]

@bwinkel - no idea, I just noticed that pytest-nunit appears in the traceback above so I tried with and without. Maybe useful to find why and fix it if these packages are useful for you, I don't know what they are doing ;).

[bwinkel]

No, I don't really need the packages. I added them long time ago and honestly, I forgot why. I was only asking because others might fall into the same trap. But it's certainly not the most urgent thing to fix...