search

astronomy-commons / hipscat-import

HiPSCat import - generate HiPSCat-partitioned catalogs
https://hipscat-import.readthedocs.io
BSD 3-Clause "New" or "Revised" License
5 stars 2 forks source link

Use scipy.sparse for histogram storage. #294

Closed delucchi-cmu closed 2 months ago

delucchi-cmu commented 2 months ago

Closes #154 using scipy.sparse for representation of healpix histogram.

Using the dense np array on-disk, the intermediate histograms for TIC are 8.5 G. Using sparse array, it's 48 M (around 200X improvement in disk usage). There's no discernible performance penalty.

codecov[bot] commented 2 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 99.52%. Comparing base (7a5c17c) to head (ad989ba).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #294 +/- ## ========================================== + Coverage 99.36% 99.52% +0.16% ========================================== Files 24 25 +1 Lines 1260 1273 +13 ========================================== + Hits 1252 1267 +15 + Misses 8 6 -2 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.