search

numba / numba-scipy

numba_scipy extends Numba to make it aware of SciPy
https://numba-scipy.readthedocs.io/en/latest/
BSD 2-Clause "Simplified" License
258 stars 34 forks source link

Bump scipy version to <1.11.0 #90

Closed je-cook closed 1 year ago

je-cook commented 2 years ago

This PR bumps scipy version to allow installation of numpy >=1.23 alongside numba and numba-scipy

As discussed in #88 the scipy upgrade was waiting for anaconda to have a later release in its default repo which it now does (https://anaconda.org/anaconda/scipy).

I have pinned to less than the next minor version of scipy as it is unlikely behaviour will change on patch version changes. If you would like this stricter let me know.

Closes #88

EDIT

Scipy have bumped the deprecation to after 1.11.0 (see https://github.com/scipy/scipy/issues/15596) and 1.10.0 is now on the anaconda repo.

david-zwicker commented 1 year ago

It would be great to merge this PR and have a new release, so we can use numba-scipy with newer scipy versions. Is there any timeline on merging this PR?

stuartarchibald commented 1 year ago

@david-zwicker I'll raise this at the public Numba meeting tomorrow. Thanks for the PR @je-cook.

je-cook commented 1 year ago

Hi, is there any feeling on when this could be merged? Its limiting our access to some f2py improvements that were added in numpy 1.23 so I'm hoping soonish :)

je-cook commented 1 year ago

gentle ping @stuartarchibald @esc is there any update on this or any chance it can be merged?

esc commented 1 year ago

gentle ping @stuartarchibald @esc is there any update on this or any chance it can be merged?

Thank you for asking about this. Perhaps @brandonwillard has some input on this too?

brandonwillard commented 1 year ago

gentle ping @stuartarchibald @esc is there any update on this or any chance it can be merged?

Thank you for asking about this. Perhaps @brandonwillard has some input on this too?

I'm all for this update, as long as the tests pass, of course (and it looks like CI is awaiting approval).

je-cook commented 1 year ago

Thanks @brandonwillard, @esc could you approve the CI run so we can move forward with this?

esc commented 1 year ago

/azp run

azure-pipelines[bot] commented 1 year ago
Azure Pipelines successfully started running 1 pipeline(s).
esc commented 1 year ago

It looks like CI is bust somehow:

==================================== ERRORS ====================================
_______________________ ERROR collecting test_sparse.py ________________________
/usr/local/miniconda/envs/test_env/lib/python3.8/site-packages/numba_scipy/tests/test_sparse.py:4: in <module>
    import scipy.sparse
/usr/local/miniconda/envs/test_env/lib/python3.8/site-packages/scipy/sparse/__init__.py:229: in <module>
    from .base import *
/usr/local/miniconda/envs/test_env/lib/python3.8/site-packages/scipy/sparse/base.py:8: in <module>
    from .sputils import (isdense, isscalarlike, isintlike,
/usr/local/miniconda/envs/test_env/lib/python3.8/site-packages/scipy/sparse/sputils.py:[16](https://github.com/numba/numba-scipy/actions/runs/4729770315/jobs/8706873616?pr=90#step:5:17): in <module>
    supported_dtypes = [np.typeDict[x] for x in supported_dtypes]
/usr/local/miniconda/envs/test_env/lib/python3.8/site-packages/scipy/sparse/sputils.py:16: in <listcomp>
    supported_dtypes = [np.typeDict[x] for x in supported_dtypes]
/usr/local/miniconda/envs/test_env/lib/python3.8/site-packages/numpy/__init__.py:320: in __getattr__
    raise AttributeError("module {!r} has no attribute "
E   AttributeError: module 'numpy' has no attribute 'typeDict'
_______________________ ERROR collecting test_special.py _______________________
ImportError while importing test module '/usr/local/miniconda/envs/test_env/lib/python3.8/site-packages/numba_scipy/tests/test_special.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/miniconda/envs/test_env/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
/usr/local/miniconda/envs/test_env/lib/python3.8/site-packages/numba_scipy/tests/test_special.py:11: in <module>
    import scipy.special as sc
/usr/local/miniconda/envs/test_env/lib/python3.8/site-packages/scipy/special/__init__.py:643: in <module>
    from .basic import *
/usr/local/miniconda/envs/test_env/lib/python3.8/site-packages/scipy/special/basic.py:[19](https://github.com/numba/numba-scipy/actions/runs/4729770315/jobs/8706873616?pr=90#step:5:20): in <module>
    from . import orthogonal
/usr/local/miniconda/envs/test_env/lib/python3.8/site-packages/scipy/special/orthogonal.py:81: in <module>
    from numpy import (exp, inf, pi, sqrt, floor, sin, cos, around, int,
E   ImportError: cannot import name 'int' from 'numpy' (/usr/local/miniconda/envs/test_env/lib/python3.8/site-packages/numpy/__init__.py)
=========================== short test summary info ============================
ERROR test_sparse.py - AttributeError: module 'numpy' has no attribute 'typeD...
ERROR test_special.py
!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!
============================== 2 errors in 10.[31](https://github.com/numba/numba-scipy/actions/runs/4729770315/jobs/8706873616?pr=90#step:5:32)s ==============================
Error: Process completed with exit code 2.

And on azure, we are seeing:

Screen Shot 2023-05-04 at 09 56 47
je-cook commented 1 year ago

Hmm odd, from what I can tell those numpy imports were removed in numpy>v1.21.

It also looks weird because I had missed the setup.py version change so I'd have thought it would actually have been the old scipy that was tested...

i've pushed the scipy change, but not quite sure how to do interdependent versioning if we need numpy<=1.21 & scipy<=1.7.3 or numpy>1.21 & scipy>1.7.3

esc commented 1 year ago

Does anyone know why I need to approve the workflow everytime?

esc commented 1 year ago

Also, does anyone know what is up with the Azure images?

je-cook commented 1 year ago

Does anyone know why I need to approve the workflow everytime?

I think its because this is my first PR to the project, on another project I've got it set to 'require approval for first time contributors' and it seems to be the case that you have to approve for all updates (for us at least)

esc commented 1 year ago

Does anyone know why I need to approve the workflow everytime?

I think its because this is my first PR to the project, on another project I've got it set to 'require approval for first time contributors' and it seems to be the case that you have to approve for all updates (for us at least)

yeah, that makes sense, I'll keep approving them until this PR is merged then.

esc commented 1 year ago

Also, does anyone know what is up with the Azure images?

could it be, that the image is outdated and now longer available?

esc commented 1 year ago

Also, does anyone know what is up with the Azure images?

could it be, that the image is outdated and now longer available?

Tryin' to bump the Azure Linux Image here: https://github.com/numba/numba-scipy/pull/95

je-cook commented 1 year ago

I forgot to quote 3.10 so it will probably fail without that

esc commented 1 year ago

Also, does anyone know what is up with the Azure images?

could it be, that the image is outdated and now longer available?

Tryin' to bump the Azure Linux Image here: #95

Seems to work, you should be able to merge this PR https://github.com/je-cook/numba-scipy/pull/1 to add my commit to this PR.

esc commented 1 year ago

So, looks like this is moving along some more now and the Azure tests are running again... but ....

https://github.com/numba/numba-scipy/pull/95#issuecomment-1534686759

je-cook commented 1 year ago

Hi @esc I've just updated everything and I think I need approval to change the macOS VM image for azure as macOS-10.15 doesnt exist like you fixed in #95.

Hopefully this time I've caught everything :slightly_smiling_face:

esc commented 1 year ago

Hi @esc I've just updated everything and I think I need approval to change the macOS VM image for azure as macOS-10.15 doesnt exist like you fixed in #95.

Hopefully this time I've caught everything 🙂

Splendid, thank you!

esc commented 1 year ago

/azp run

azure-pipelines[bot] commented 1 year ago
Azure Pipelines successfully started running 1 pipeline(s).
esc commented 1 year ago

@brandonwillard can I get a second set of eys 👀 on this? it LGTM from my end.

esc commented 1 year ago

@je-cook good news, this one tested as green.

je-cook commented 1 year ago

Looks like github was not as clever as I hoped and if you leave out a variable in an include outside of the matrix it defaults to an empty string and doesnt cycle through the matrix for that value!

esc commented 1 year ago

OK, looks like the test matrix is all green, However, I see that there are tests for Python 3.6 and 3.7 -- now I am confused as to why we want these? The latest Numba release only supports 3.8, 3.9, 3.10 and 3.11..

See also: the version support table for Numba.

https://numba.readthedocs.io/en/stable/user/installing.html#version-support-information

je-cook commented 1 year ago

Happy to drop them if we're keeping in lockstep with numba itself. I just didn't want to deprecate anything without wider consent

je-cook commented 1 year ago

I'm not sure whether the 32bit version will work with the newer stuff but we'll see!

esc commented 1 year ago

I'm not sure whether the 32bit version will work with the newer stuff but we'll see!

I don't think we support 32-bit anymore. But I am not certain as of when that's the case. @stuartarchibald do you know if we have a hardware-support table too?

esc commented 1 year ago

@je-cook what's up with the Numpy 1.24 on conda issue?

je-cook commented 1 year ago

I think its because numba 0.57 (which added numpy 1.24 support) is not on the conda repo yet. Thats my best guess looking around the azure logs for a few minutes, https://dev.azure.com/numba/numba-scipy/_build/results?buildId=15042&view=results

esc commented 1 year ago

I think its because numba 0.57 (which added numpy 1.24 support) is not on the conda repo yet. Thats my best guess looking around the azure logs for a few minutes, https://dev.azure.com/numba/numba-scipy/_build/results?buildId=15042&view=results

Yes, looking at:

https://anaconda.org/conda-forge/numba/files

I concur with this assessment. I would like to find out when 0.57 is likely to land in CF and then decide on how to proceed.

esc commented 1 year ago

I think its because numba 0.57 (which added numpy 1.24 support) is not on the conda repo yet. Thats my best guess looking around the azure logs for a few minutes, https://dev.azure.com/numba/numba-scipy/_build/results?buildId=15042&view=results

Yes, looking at:

https://anaconda.org/conda-forge/numba/files

I concur with this assessment. I would like to find out when 0.57 is likely to land in CF and then decide on how to proceed.

Looking at:

https://github.com/conda-forge/llvmlite-feedstock/pull/69

and

https://github.com/conda-forge/numba-feedstock/pull/115

it may been some time until 0.57.0 lands in conda-forge -- thus I would suggest to merge this PR as it get's the job done and then add in 1.24 when 0.57.0 lands on CF. Are well good with that?

esc commented 1 year ago

@je-cook thank you for the patch! @brandonwillard thank you for the review!

t-brito commented 1 year ago

Apologies for posting in a Merged PR, but I was wondering if there were any plans to package this change into a new patch version on PyPI (i.e. version 0.3.2?)

esc commented 1 year ago

Apologies for posting in a Merged PR, but I was wondering if there were any plans to package this change into a new patch version on PyPI (i.e. version 0.3.2?)

Feel free to open a new release request as an issue on github and I'll see what I can do about a release. 👍