desihub / desispec

DESI spectral pipeline
BSD 3-Clause "New" or "Revised" License
33 stars 24 forks source link

parallelize zpix check #2275

Closed sbailey closed 2 weeks ago

sbailey commented 1 month ago

This PR refactors desi_resubmit_zpix to parallelize the file checking for dramatically faster runs. It also prints all of the TODO items at the end instead of having them interspersed with N>>1 other log messages.

This PR is useful for Jura, but not urgent since we can run --dry-run outside of the jura env and it doesn't impact any actual processing.