astropy / reproject

Python-based Astronomical image reprojection :milky_way: - maintainer @astrofrog
https://reproject.readthedocs.io
BSD 3-Clause "New" or "Revised" License
105 stars 64 forks source link

Improve performance for large datasets and switch to multi-threading by default #443

Closed astrofrog closed 3 weeks ago

astrofrog commented 1 month ago

This is a work in progress to address serious performance issues when reprojecting large datasets, mainly with reproject_interp.

To do:

Fixes #393 Fixes #394

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 81.21547% with 34 lines in your changes missing coverage. Please review.

Project coverage is 91.57%. Comparing base (38b134b) to head (7f7f90b). Report is 3 commits behind head on main.

Files Patch % Lines
reproject/mosaicking/coadd.py 74.78% 29 Missing :warning:
reproject/array_utils.py 91.07% 5 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #443 +/- ## ========================================== - Coverage 93.54% 91.57% -1.97% ========================================== Files 25 25 Lines 929 1021 +92 ========================================== + Hits 869 935 +66 - Misses 60 86 +26 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

astrofrog commented 4 weeks ago

Running into memory issues when mosaicking large datasets after the first one is done which suggests maybe an issue with the data getting loaded at the point where it is added to the main array.

astrofrog commented 3 weeks ago

Some CI failures related to zarr 3.0.0a1 released yesterday, need to investigate.

astrofrog commented 3 weeks ago

I'll go ahead and merge this and will then check the multi-processing and document any steps needed in a separate PR.