pytroll / pyresample

Geospatial image resampling in Python
http://pyresample.readthedocs.org
GNU Lesser General Public License v3.0
343 stars 95 forks source link

Fix Cython 3 and Numpy 2 compatibility #582

Closed djhoese closed 5 months ago

djhoese commented 5 months ago

TODO: I should update wheel building to use numpy 2 when creating wheels so future wheels will already be compatible.

codecov[bot] commented 5 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (f892115) 94.06% compared to head (fa3771b) 93.99%. Report is 5 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #582 +/- ## ========================================== - Coverage 94.06% 93.99% -0.07% ========================================== Files 90 92 +2 Lines 13590 13777 +187 ========================================== + Hits 12783 12950 +167 - Misses 807 827 +20 ``` | [Flag](https://app.codecov.io/gh/pytroll/pyresample/pull/582/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pytroll) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/pytroll/pyresample/pull/582/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pytroll) | `93.99% <100.00%> (-0.07%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pytroll#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

coveralls commented 5 months ago

Coverage Status

coverage: 93.656%. remained the same when pulling fa3771b38e7fc3dfc37a0b968bd9752ccd0ba43e on djhoese:feat-cython3 into 3887db64cb82e17fc3c92114bb8078d9bc98190b on pytroll:main.

djhoese commented 5 months ago

Let's see if I can get all my profiling results for gradient search organized. These are all doing ABI C02 (500m resolution) full disk to a large eqc target area and the final compute is going to nowhere so there is no to-disk writing involved in this. Only 4 dask workers involved. Chunk size set via PYTROLL_CHUNK_SIZE to 1356 but I think that still rounds up.

Summary

My indexing changes are negligible in this use case. They were maybe a bit faster (~2 seconds of 47) but to me that's nothing. Both newer Cython 3 versions (the first 2 results below) are about the same as main on Cython 2. I'll try more workers in a second to see how it goes.

In reverse chronological/development order:

This PR

image

This PR but without the new column indexing

image

Main

Note: I had to force Cython <3 to be used here otherwise it took 6+ minutes to run because it was hitting the GIL a ton. We need to get this PR merged and released as soon as possible.

image

djhoese commented 5 months ago

Yeah no differences with 8 workers over 4 workers.

djhoese commented 5 months ago

As mentioned on slack the unstable CI environment is failing due to upstream libraries like shapely and rasterio not being compatible with newest numpy 2 changes. I'll merge this PR as-is and then make another PR to test against my own custom fork of these libraries with fixes.

djhoese commented 5 months ago

Just realized that I never updated CI to build wheels against numpy 2. I'll do that in another PR.