search

LSSTDESC / SSim_DC1

Configuration, production, validation specifications and tools for the DC1 Data Set.
4 stars 2 forks source link

Modify DM workflow for DC1 production #9

Closed cwwalter closed 7 years ago

cwwalter commented 8 years ago

The current CI workflow needs to be tweaked to work at NERSC. We need to understand how to make both the workflow and DM work on full focal planes and more generally how to run in the time restricted batch environment there.

drphilmarshall commented 8 years ago

I think the workflow engine may not need modifying, but the simulation and DM processing workflows will do. I guess this issue is mainly about the DM Level 2 pipeline, right? Or are you including the simulation pipeline as well? It makes sense to keep the two separate, we found.

richardxdubois commented 8 years ago

Strictly speaking, how we end up running at NERSC could drive changes to the workflow engine. At present it could not handle wrapping jobs in the MPI queue. If the solution is to bulk up the shared queue, we're good to go already.

cwwalter commented 8 years ago

Two issues mostly as far as I understood @richardxdubois and @tony-johnson should comment more or assign themselves to this issue:

1) DM pipeline. This is related to #7.

2) Really making everything run at NERSC is an issue related to queues etc.

cwwalter commented 8 years ago

Oops crossed messages. Yes, exactly as Richard says.

jchiang87 commented 8 years ago

With regard to

1) DM pipeline. This is related to #7.

I think we will want to modify any tasks in the Twinkles Level 2 pipeline having to do with the coadds to loop in parallel over "patches" and modify the launching of the forced photometry tasks to loop over sensors as well as over visits. In working on #7, I wrote some tools to help with these.

cwwalter commented 7 years ago

The workflow engines were modified in order to run DC1 and this is a continuing topic of work and exploration for the CI group.