astronomy-commons / hipscat-import

HiPSCat import - generate HiPSCat-partitioned catalogs
https://hipscat-import.readthedocs.io
BSD 3-Clause "New" or "Revised" License
5 stars 2 forks source link

Pickle instead of scatter. #328

Closed delucchi-cmu closed 1 month ago

delucchi-cmu commented 1 month ago

Change Description

Closes #302.

Writes the file reader to a pickled file, and de-pickles in the mapping and splitting workers.

Writes the alignment to a pickled file, and de-pickles in the splitting workers. destination map is created on-demand, and is owned by the resume plan, as this is used to determine remaining reduce tasks.

Code Quality

codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 99.78%. Comparing base (c5e6095) to head (a66e111).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #328 +/- ## ========================================== + Coverage 99.56% 99.78% +0.21% ========================================== Files 26 26 Lines 1379 1389 +10 ========================================== + Hits 1373 1386 +13 + Misses 6 3 -3 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.