Sbpy in conda environment causes pytest collection to fail in Python<3.11

[astronomerritt]

Creating a conda environment with sbpy causes pytest to fail to collect every single test due to a RuntimeWarning in Python<3.11.

The environments are created via:

conda create -n sspp_test -c conda-forge -c mjuric python=3.10 spiceypy=4.0.1 openorb numpy pandas matplotlib spice-utils pip pytest astropy scipy sbpy pytables

Then running pytest on this repo produces the following error during test collection for every single test:

____________________________________________________ ERROR collecting surveySimPP/modules/tests/test_PPVignetting.py ____________________________________________________
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1050: in _gcd_import
    ???
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:992: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1050: in _gcd_import
    ???
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:992: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1050: in _gcd_import
    ???
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:992: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1050: in _gcd_import
    ???
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:1006: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:688: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:883: in exec_module
    ???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
    ???
surveySimPP/__init__.py:1: in <module>
    from . import modules
surveySimPP/modules/__init__.py:3: in <module>
    from . import PPCalculateApparentMagnitude
surveySimPP/modules/PPCalculateApparentMagnitude.py:3: in <module>
    from .PPCalculateApparentMagnitudeInFilter import PPCalculateApparentMagnitudeInFilter
surveySimPP/modules/PPCalculateApparentMagnitudeInFilter.py:6: in <module>
    from sbpy.photometry import HG, HG1G2, HG12_Pen16, LinearPhaseFunc
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/sbpy/photometry/__init__.py:5: in <module>
    from .core import *
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/sbpy/photometry/core.py:16: in <module>
    from astropy.modeling import (Fittable1DModel, Parameter)
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/modeling/__init__.py:10: in <module>
    from . import fitting, models
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/modeling/fitting.py:39: in <module>
    from .spline import (  # noqa: F401
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/modeling/spline.py:15: in <module>
    from .core import FittableModel, ModelDefinitionError
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/modeling/core.py:29: in <module>
    from astropy.nddata.utils import add_array, extract_array
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/nddata/__init__.py:14: in <module>
    from .blocks import *
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/nddata/blocks.py:8: in <module>
    from .decorators import support_nddata
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/nddata/decorators.py:12: in <module>
    from .nddata import NDData
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/nddata/nddata.py:12: in <module>
    from astropy.wcs.wcsapi import (
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/wcs/__init__.py:26: in <module>
    from . import utils
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/wcs/utils.py:8: in <module>
    from astropy.coordinates import ITRS, CartesianRepresentation, SphericalRepresentation
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/coordinates/__init__.py:13: in <module>
    from .builtin_frames import *
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/coordinates/builtin_frames/__init__.py:30: in <module>
    from .cirs import CIRS
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/coordinates/builtin_frames/cirs.py:8: in <module>
    from .utils import DEFAULT_OBSTIME, EARTH_CENTER
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/coordinates/builtin_frames/utils.py:15: in <module>
    from astropy.utils import iers
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/utils/iers/__init__.py:1: in <module>
    from .iers import *
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/utils/iers/iers.py:23: in <module>
    from astropy.table import MaskedColumn, QTable
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/table/__init__.py:6: in <module>
    from .column import Column, ColumnInfo, MaskedColumn, StringTruncateWarning
/opt/miniconda3/envs/sspp_test_310/lib/python3.10/site-packages/astropy/table/column.py:20: in <module>
    from ._column_mixins import _ColumnGetitemShim, _MaskedColumnGetitemShim
astropy/table/_column_mixins.pyx:1: in init astropy.table._column_mixins
    ???
E   RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 96 from PyObject

As can be seen, the traceback leads to sbpy and then to astropy.

Tests run as expected in Python 3.11, no warnings. There's a full writeup in this issue.

For both Python 3.10 and Python 3.11: Operating system and version: MacOS Monterey 12.6.2 sbpy version: 0.3.0 astropy version: 5.2.1 numpy version: 1.24.1

Thanks in advance for any help!

[astronomerritt]

Additional important note: pip-installing sbpy does not cause this problem to occur.

[mkelley]

When I search on this error, I see a lot of references to upgrading or reinstalling numpy. Does pip installing sbpy affect the numpy version or does it pip install another numpy?

[astronomerritt]

No, the numpy version remains the same: 1.24.1.

The pip install of sbpy only installs the following:

Successfully installed MarkupSafe-2.1.2 ads-0.12.3 astroquery-0.4.6 beautifulsoup4-4.11.1 html5lib-1.1 httpretty-0.8.10 jaraco.classes-3.2.3 keyring-23.13.1 mock-5.0.1 more-itertools-9.0.0 pyvo-1.4 sbpy-0.3.0 soupsieve-2.3.2.post1 synphot-1.1.1 webencodings-0.5.1 werkzeug-2.2.2

[astronomerritt]

If it's any help, the conda install does work with no test collection failure for Python 3.6 and sbpy 0.2.2: this installs numpy 1.19.5 and astropy 3.2.3. However the issue resumes with Python 3.7 (sbpy 0.3.0, numpy 1.21.6, astropy 4.3.1).

[astronomerritt]

I'm still having this issue - any advice? We're looking to release soon, but we're unwilling to do so unless we can trust that a full conda environment will work without issues.

[mkelley]

@astronomerritt Apologies for missing this message! I do have a PR that updates a few things that might help. We'll get that merged ASAP.

[mkelley]

Although it may be we need to do a release, since you are installing via conda. That will still take a couple weeks.

But maybe re-building the conda package could address this? @mwcraig , is this something that can be done without a release? And if there is something that I can do to make it happen, let me know.

[mkelley]

Ah, got it. I'm seeing now that astropy ignores this warning. If we do the same, then the tests should pass. We will need to make a new release.

[mkelley]

@astronomerritt In the meantime, are you able to add command line arguments to filter the warning out? Something like -W ignore:numpy.ndarray size changed:RuntimeWarning The documentation on the warnings filter is at https://docs.python.org/3/library/warnings.html#warning-filter

[astronomerritt]

This is terrific news, thanks for getting back to this! We're absolutely fine to wait a couple weeks for the next conda release on this, our own release has been delayed, as these things often are.

[mwcraig]

@mkelley -- we could patch the source in the conda recipe as a temporary workaround but since a new release is coming soon it seems better to wait.