Investigate: Allow setting fast-track processing

[vinjana]

In PBS this was implemented via accountName and -A option (might be specific for our cluster though).

• How is this implemented in LSF? Via queue(s)?
• How can we implement this? ResourceSetSize?
• Note that queue and priority are independent concepts. If there are convey and convey_fast queues as in the PBS cluster, this may mix concepts.

[vinjana]

The first implementation of fast-track in the LSF cluster will be via queues. There is a "fasttrack" queue (see bqueues command on head node) and similar queue are planned for "highmem" and "gpu".

As we need different such resources and therefore different queues for different jobs (running sambamba on the FPGA node doesn't make sense), we need a way to select different sets of resources on the Roddy command line.

The solution we take is based on the existing feature that resource sets can be selectively overwritten in the XMLs.

• We will add a --additionalImports parameter, which will be used similar to --additionalImports=pbs_fasttrack or --additionalImports=lsf_fasttrack (needs to be implemented in Roddy).
• Additional configurations with the names used in the --additionalImports parameter need to be in the configuration directories, e.g. in extra files, such as "pbs_fasttrack.xml" or "lsf_fasttrack.xml". These must contain the configurations names: <configuration name="pbs_fasttrack" ...> used in the --additionalImports parameter.
• Each cluster-specific configuration will contain a set of cluster specific resource set definitions for all cluster jobs. That is, e.g. for running the alignment job on the LSF cluster may have

  <tool name='alignAndPairSlim'  value='bwaMemSortSlim.sh' basepath='qcPipeline' overrideresourcesets="true">
  <resourcesets>
     <rset size="t" memory="12" cores="8" nodes="1" walltime="00:10:00" queue="fpga"/>
     <rset size="s" memory="12" cores="8" nodes="1" walltime="2" queue="fpga"/>
     <rset size="m" memory="17" cores="8" nodes="1" walltime="6" queue="fpga"/>
     <rset size="l" memory="20" cores="8" nodes="1" walltime="60" queue="fpga"/>
     <rset size="xl" memory="20" cores="8" nodes="1" walltime="160" queue="fpga"/>
  </resourcesets>
  </tool>
  <tool name='mergeAndRemoveDuplicatesSlimSambamba' value='mergeAndMarkOrRemoveDuplicatesSlim.sh' basepath='qcPipeline' overrideresourcesets="true">
   <resourcesets>
       <rset size="t" memory="30" cores="6" nodes="1" walltime="00:10:00" queue="devel"/>
       <rset size="xs" memory="3" cores="6" nodes="1" walltime="1" queue="fasttrack"/>
       <rset size="s" memory="4" cores="6" nodes="1" walltime="15" queue="fasttrack"/>
       <rset size="m" memory="5" cores="6" nodes="1" walltime="20" queue="fasttrack"/>
       <rset size="l" memory="6" cores="6" nodes="1" walltime="30" queue="fasttrack"/>
       <rset size="xl" memory="8" cores="6" nodes="1" walltime="80" queue="fasttrack"/>
   </resourcesets>
  </tool>

  The resource set overriding is the already implemented behaviour. Note that

  1. For all cluster job type (or "tool") that should be overridden the "tool" block is given.
  2. The tool block has the attribute overrideresourcesets="true" (otherwise also the I/O files are overwritten)
  3. Currently, the tool block attributes must include the value and basepath attributes (otherwise this information is not available anymore)
  4. The only information within the tool block is the "resourcesets" block. The block will overwrite the original configuration en bloc.
  5. Therefore, here all resource set sizes (rset size="...") are given. Resource set size that are not given in this cluster-specific XML will not be available (i.e. the resource sets in the workflow XML are lost).

The only feature that needs to be added to Roddy is the --additionalImports parameter that allows to add imported configurations' names to the project-configuration.

[vinjana]

Solved with #111.