uclahs-cds / pipeline-align-DNA

Nextflow pipeline to align paired DNA FASTQs and sort, mark duplicates, and index resulting alignment
https://uclahs-cds.github.io/pipeline-align-DNA/
GNU General Public License v2.0
4 stars 1 forks source link

Use `methods.setup_process_afterscript()` for process logs #321

Closed nwiltsie closed 5 months ago

nwiltsie commented 5 months ago

We should incorporate the new methods.setup_process_afterscript() from https://github.com/uclahs-cds/pipeline-Nextflow-config/pull/64 into this pipeline.

Under the new method, log files are automatically saved to ${params.log_output_dir}/process-log/${task.ext.log_dir}${task.ext.log_dir_suffix}/ (no need to specify a publishDir directive).

${task.ext.log_dir} defaults to ${task.process.replace(':', '/')}. ${task.ext.log_dir_suffix} defaults to a blank string.

The key steps are:

  1. Add methods.setup_process_afterscript() to methods.setup().
  2. For each Nextflow process:
    1. Remove the file ".command.*" output directive.
    2. Add ext log_dir { ... } and ext log_sur_suffix { ... } directives, if appropriate.
    3. Remove the publishDir directive with pattern: ".command.*".

As an example, this publishDir directive... https://github.com/uclahs-cds/pipeline-align-DNA/blob/05275a7c81c9077e7f5fa41e48c7156494edfbdb/module/align_DNA_BWA_MEM2.nf#L35-L38

... should be replaced with this directives:

ext log_dir { "${params.bwa_version}/${task.process.split(':')[1].replace('_', '-')}" }
ext log_dir_suffix { "/${library}/${lane}" }

See https://github.com/uclahs-cds/pipeline-recalibrate-BAM/pull/76/files for a full example.

j2salmingo commented 5 months ago

If this is a boilerplate help template that you are copying into other pipelines:

2. For each Nextflow process:

   1. Remove the `file ".command.*"` `output` directive.
   2. Add `ext log_dir { ... }` and `ext log_sur_suffix { ... }` directives, if appropriate.
   3. Remove the `publishDir` directive with `pattern: ".command.*"`.

Without the colon, stdout will log an error but Nextflow will not exit with an error code.