astronomy-commons / hipscat-import

HiPSCat import - generate HiPSCat-partitioned catalogs
https://hipscat-import.readthedocs.io
BSD 3-Clause "New" or "Revised" License
5 stars 2 forks source link

Reduce stage should fail faster if any individual task fails #300

Closed delucchi-cmu closed 1 month ago

delucchi-cmu commented 2 months ago

Bug report

The wait_for_futures method will report individual failures, but for the reduce step, we should fail immediately on the first failure, as there's little point in continuing on if some pieces fail.

Before submitting Please check the following: