Spin-off the DM Level 2 workflow engine code in Twinkles into a separate package

[jchiang87]

@tony-johnson The workflow described at the ReprocessingTaskForce wiki is very similar to the "Level 2" workflow described in the Twinkles cookbook. We should spin off the workflow engine code in Twinkles and generalize it so that it can be used to execute various similar analysis pipelines in a configurable way.

[drphilmarshall]

I guess the pipeline code would live in a repo called "DMLevel2" or something? And then we'd need it to be able to be run by either the workflow engine, or by hand (in order to support the Reprocessing Group well). @boutigny what do you think about this notion, of having only one emulation of the DM Level 2 pipeline, that both Twinkles and Reprocessing uses? Any high level thoughts before we start down this road?

[jchiang87]

I think we'd like to make it such that this new workflow engine-oriented package could run pipelines other than just the versions of Level 2 used by Twinkles or the Reprocessing Task Force.

[boutigny]

Having one emulation of the DM Level 2 pipeline that can be used for Twinkles, the Reprocessing task force or any similar applications is a good thing as long as it provides all the necessary flexibility to accept any configuration parameters (I guess that this is straightforward) and is able to run on any reasonable computing facility.

[tony-johnson]

@johannct may be interested in helping with this, as well as with getting the workflow able to run at IN2P3.

[heather999]

Looks like Johann was interested, so much so that he ran DC2 processing at IN2P3