search

glue-viz / glue

Linked Data Visualizations Across Multiple Files
http://glueviz.org
Other
721 stars 152 forks source link

Compatibility and tests against Numpy 2.0.0rc1 #2471

Closed dhomeier closed 2 months ago

dhomeier commented 6 months ago

Description

Installing numpy dev in CI is currently blocked by various upstream packages pinned to numpy < 2.0; adding those to the developer versions. This would update both dev jobs to numpy>=0.0.dev0; restrict to a new env npdev instead?

Only workaround for the pins seems to be to install with --no-deps (pandas-dev/pandas#55488) which in turn requires every single test dependency to be pulled in explicitly; if anyone has an idea how to run a first pass without --no-deps an then a second pip install to override the numpy pin, that could be simplified a bit.

Building shapely seems to require compiling its own libgeos installation per its install script; unless I find a way to get an apt install for that, this is a showstopper for now. (It seems to be available, but would require pointing the CI to fetch from an alternative repo).

pllim commented 6 months ago

For pandas and matplotlib, I did a hack in my packages to let them downgrade numpy first but then force numpy dev reinstall right before test start running and ignore the pip warning. It's not pretty but easier than --no-deps .

shapely is indeed more troublesome. They never replied to my request.

dhomeier commented 4 months ago

@pllim & @jorisvandenbossche thanks for following up on the shapely update; this is now working with the wheel from scientific-python-nightly although I still needed to put a hard pin on the version until a 2.1.0.dev0 is added in shapely/shapely#1999 – so the revision might not be very durable.

@astrofrog I think the numpy<2.0 pin on casa-formats-io could be removed as the main branch installed with numpy-dev without problems (though maybe needs to be tested on more platforms and Python versions); this is the only devdep left forcing us to install with --no-deps and all the resultant hassle. And of course would be good to have numpy-2.0 wheels for that and fast_histogram at some point.

dhomeier commented 4 months ago

Starting to test Python 3.12 as well addressing #2470; Windows data_factory confusion to be investigated. Update: same failure with 3.11; probably the reason I had already removed the Windows py311 job.

jorisvandenbossche commented 4 months ago

I just uploaded wheels that should have the higher version number (https://anaconda.org/scientific-python-nightly-wheels/shapely/files), that should fix the issue with having to pin

dhomeier commented 4 months ago

I just uploaded wheels that should have the higher version number (https://anaconda.org/scientific-python-nightly-wheels/shapely/files), that should fix the issue with having to pin

Yes, everything works with >=0.0.dev0 now. Thanks again!

astrofrog commented 2 months ago

Just FYI I'm working on casa-formats-io (https://github.com/radio-astro-tools/casa-formats-io/pull/61) and fast-histogram is currently building wheels for a new Numpy 2.0-compatible release.

dhomeier commented 2 months ago

Running now with numpy 2.0.0rc1, astropy 6.1.0rc1 and scipy 1.13.0; fast-histogram still needs its own build (see https://github.com/astrofrog/fast-histogram/pull/86#issuecomment-2059600339); have not checked for released versions of other dependencies yet. The actual glue test failures all seem to be on np.array(copy=False) like https://github.com/glue-viz/glue/actions/runs/8709773487/job/23890261129?pr=2471#step:10:311 The fix looks straightforward enough, going to look into that next.

dhomeier commented 2 months ago

@astrofrog I have used a simplified version of the Scipy fix, since we should not need to worry about earlier numpy 2.0.0dev versions that did not support copy=None yet. The new casa-formats-io wheel on PyPI still depends on and was built against numpy 1.26 – I could not follow the cibuildwheel process entirely, but it is also testing against that: https://github.com/radio-astro-tools/casa-formats-io/actions/runs/8720397145/job/23921900123#step:7:371 perhaps need to remove the python_version<'3.9' case from pyproject.toml altogether?

Shapely has a new release built on shapely/shapely#2033 but seems to suffer the same problems with wheels for older Pythons being pulled in. – may just have had some delays in uploading all wheels, so for all other dependencies the current releases or candidates now work. 🚀

Before merging we should decide which 0.0.dev0 versions to keep for testing in the dev env (numpy, scipy, astropy?)

Don't have a way to debug the pandas-related Windows failures.

astrofrog commented 2 months ago

@dhomeier - are you sure it's still not working with casa-formats-io v0.3.0? There is a devdeps job there that shows that it works fine with numpy 2.0.0rc1 at runtime: https://github.com/radio-astro-tools/casa-formats-io/actions/runs/8720397149/job/23921906624

dhomeier commented 2 months ago

https://github.com/glue-viz/glue/actions/runs/8723259903/job/23931144239?pr=2471 looks OK now

astrofrog commented 2 months ago

@dhomeier is this ok to merge? (If so feel free to!)

dhomeier commented 2 months ago

Yes, think so – I made on run with pandas dev to check the Windows failures (still persisted), but don't think we need that in general; thanks!