mskcc / tempo

CCS research pipeline to process WES and WGS TN pairs
12 stars 5 forks source link

nextflow updates causing errors in execution #978

Open anoronh4 opened 1 year ago

anoronh4 commented 1 year ago
  1. trace file rolling is now disabled in newer versions of nextflow: https://github.com/nextflow-io/nextflow/issues/3317. With our current configuration trace { file = 'trace.txt' }, there are errors with resume because nextflow is attempting to write to the same location without rolling the previous file (unless we specify overwrite = true). Similar applies to execution report and execution timeline. We should change how we output trace files and possibly adjust tempodeliver to expect trace files with different names.
  2. nextflow sometimes ignores config directives written with the dot notation (e.g. process.errorStrategies rather than process {}), documented at https://github.com/nextflow-io/nextflow/issues/3154. The developers said they plan to address it but i am still finding the issue in a recent release (22.10.3). We should put dot notation configs into braces {} to avoid issues going forward.
gongyixiao commented 1 year ago

Both issues were not present in nextflow 21.10.6 or earlier.

anoronh4 commented 1 year ago

for fixing how the trace file is written, i wonder if it would make sense to use an implementation such as the following:

https://github.com/mskcc/forte/blob/b556deb986ac6df217c4f2b0c6536794c7a69d4f/nextflow.config#L50-L51

https://github.com/mskcc/forte/blob/b556deb986ac6df217c4f2b0c6536794c7a69d4f/nextflow.config#L167-L171

(not sure why the code preview isn't showing up) basically where all execution reports go to a named, unique folder each time the run initializes. that way we can save all trace files and have them be organized.

gongyixiao commented 1 year ago

I think this should work fine.