astronomy-commons / hipscat-import

HiPSCat import - generate HiPSCat-partitioned catalogs
https://hipscat-import.readthedocs.io
BSD 3-Clause "New" or "Revised" License
5 stars 2 forks source link

Fail fast on reduce stage failures. #317

Closed delucchi-cmu closed 1 month ago

delucchi-cmu commented 1 month ago

Change Description

Closes #300.

Adds code path for the waiting for futures operation to re-raise exception on the first exception encountered. The reducing stage is a little unique, in that a failure in some shard indicates that the whole pipeline may be experiencing issues, whereas failures at mapping and splitting are more likely to indicate a problem with an individual file.

Code Quality

Project-Specific Pull Request Checklists

New Feature Checklist

codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 99.52%. Comparing base (37ce248) to head (63e8684).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #317 +/- ## ======================================= Coverage 99.52% 99.52% ======================================= Files 25 25 Lines 1270 1272 +2 ======================================= + Hits 1264 1266 +2 Misses 6 6 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.