search

hiddenSymmetries / simsopt

Simons Stellarator Optimizer Code
https://simsopt.readthedocs.io
MIT License
83 stars 43 forks source link

Numpy2.0 migration #427

Open smiet opened 1 month ago

smiet commented 1 month ago

The container build is failing because the SPEC and VMEC2000 compilation step are failing.

The migration guide to numpy2.0 is given here.

I will work on getting SPEC up and running with numpy 2.0, @landreman can you have a look at VMEC? from this link it looks like this is what should be added: 

[build-system]
build-backend = ...
requires = [
    # Note for packagers: this constraint is specific to wheels
    # for PyPI; it is also supported to build against 1.xx still.
    # If you do so, please ensure to include a `numpy<2.0`
    # runtime requirement for those binary packages.
    "numpy>=2.0.0rc1",
    ...
]

[project]
dependencies = [
    "numpy>=1.23.5",
]
landreman commented 1 month ago

Hm, I tried that change in the vmec2000 repo, but there are various errors in the vmec2000 CI now. Looping in @mbkumar @andrewgiuliani @daringli @rogeriojorge in case anyone can help.

smiet commented 1 month ago

Now thinking we might want to temporarily force numpy<2.0 in all runners, there are other dependencies like f90wrap that use the deprecated oldest-supported-numpy that need to be patched, as well as that a new pyoculus needs to be uploaded to pypi as we resolve all the dependencies

smiet commented 1 month ago

Update: f90wrap needs to be adapted as well to make spec (and simsopt) run with numpy 2.0. Working with the upstream to merge a pull-request.

In the meantime we can try to force the runners to use a 1.x version of numpy, as I tried in PR #429

Dependencies that still need to be patched:

smiet commented 4 weeks ago

nptyping does not seem to have any compatibility with numpy2.0, and when installed under 2.0 raises an exception upon import. Currently it's dependency is set to numpy<2.0, such that installing simsopt will downgrade numpy to 1.26.

Can we replace the dependency? @mbkumar, I see you added it, what does nptyping allow us to do that numpy.typing cannot provide?

mbkumar commented 4 weeks ago

nptyping was the first to come out with type hinting for numpy data structures. It was added more than couple of years ago and I haven't kept up to date with the progress made by numpy in supporting type hinting. I think this is a good time to look at the type hinting landscape and see if we still need to use nptyping library.

mbkumar commented 4 weeks ago

Looks like development on nptying ceased last year. We should switch to numpy.typing. I'll work on it ASAP.

smiet commented 3 weeks ago

Tests are failing because of a few deprecated numpy uses. See the branch ml/vmec_update_for_numpy_2 and it's tests for progress.

@florianwechsung the randomgen package is not updated for numpy 2.0, could np.random be used, or is it essential for the stochastic optimizations?

mbkumar commented 3 weeks ago

@smiet What is the status of f90wrap? Do we still need to use CZhu's repo? His repo is not passing python 3.11 test.

smiet commented 3 weeks ago

@mbkumar You need to use the main_off branch, this is up-to-date with remote, works with numpy2.0 and has an open pull request.

For now this branch is needed for the SPEC installation yes, the upstream branch fails because variables contain python keywords.

smiet commented 3 weeks ago

Sorry @mbkumar, the new branch also needs (for the time being) the numpy2_fix branch of SPEC. The PR is awaiting review.

See the changes I made the ml/vmec_update_for_numpy_2 workflows

    - name: Download SPEC
      run: git clone -b numpy2_fix --depth=1 https://github.com/PrincetonUniversity/SPEC

Hopefully this PR will be resolved tomorrow.

mbkumar commented 3 weeks ago

@smiet Thanks for letting me know. I'll wait till the PRs are merged. Please ping me once the PRs are merged.

smiet commented 3 weeks ago

@mbkumar You are a contributor to SPEC, If you give the PR a quick glance and agree with the changes, you can approve and merge.

smiet commented 3 weeks ago

@mbkumar The new version of SPEC is merged into master

florianwechsung commented 3 weeks ago

Tests are failing because of a few deprecated numpy uses. See the branch ml/vmec_update_for_numpy_2 and it's tests for progress.

@florianwechsung the randomgen package is not updated for numpy 2.0, could np.random be used, or is it essential for the stochastic optimizations?

I vaguely recall using randomgen because the standard numpy.random does not satisfy certain independence conditions when running in parallel. I'm travelling this week but will have a look on the weekend. Please ping me again if I don't come back to you by Monday.