Use lessons learned during DR2 pass 1 processing to harden/refactor/update the workflow for the (gen2) processing workflow using Parsl
Contacts: Tom Glanzman, Ben Clifford
Day/Time: Tue-Fri, mostly asynchronous due to time zone difference
Main communication channel: slack/zoom (details TBA)
GitHub repo: LSSTDESC ImageProcessingPipelines
Goals and deliverable
A more robust workflow with better bookkeeping and introspection
Resources and skills needed
python, parsl, DM command line tasks (desired), NERSC environment
Detailed description
Parsl is the software system used to organize and execute a sequence of DM tasks on a large dataset. The DR2 (pass 1) dataset is the first production attempt to use Parsl in this problem domain. There were a number of problems encountered and this sprint is an attempt to address some of these problems.
Some possible goals:
Integration of latest monitoring fixes and enhancements
Standardize log and metadata files and reorganize their directory structure
Inspect and interpret terminal futures
Design DM command line task wrapper to allow pre- and post- task checks
Systematically add better log messages to tie parsl (run/task) actions and bookkeeping to DM processing (tract/patch/filter/visit/sensor/raft)
Rethink log level messages to significantly reduce uninteresting traffic
Design scheme for handling the need to redo or retry processing campaign subsets
DR2 Parsl workflow hardening
Use lessons learned during DR2 pass 1 processing to harden/refactor/update the workflow for the (gen2) processing workflow using Parsl
Contacts: Tom Glanzman, Ben Clifford Day/Time: Tue-Fri, mostly asynchronous due to time zone difference Main communication channel: slack/zoom (details TBA) GitHub repo: LSSTDESC ImageProcessingPipelines
Goals and deliverable
A more robust workflow with better bookkeeping and introspection
Resources and skills needed
python, parsl, DM command line tasks (desired), NERSC environment
Detailed description
Parsl is the software system used to organize and execute a sequence of DM tasks on a large dataset. The DR2 (pass 1) dataset is the first production attempt to use Parsl in this problem domain. There were a number of problems encountered and this sprint is an attempt to address some of these problems. Some possible goals: