WARN: Process 'ncovIllumina:sequenceAnalysis:writeQCSummaryCSV' cannot be executed by 'google-lifesciences' executor -- Using 'local' executor instead

[mikek498]

I am getting a "WARN: Process 'ncovIllumina:sequenceAnalysis:writeQCSummaryCSV' cannot be executed by 'google-lifesciences' executor -- Using 'local' executor instead" when running the pipeline against the Google Life Sciences API.

The workflow does not complete successfully. Any help would be appreciated.

[m-bull]

That WARN is expected and won't (shouldn't) cause the workflow to fail - the use of multiple executors is supported by Nextflow.

I'll need more information on how the workflow fails if you want me to debug this further - you might be able to include a scrubbed trace from .nextflow.log.

[mikek498]

It running locally does not allow it to succeed. @mjpworth can you please provide the .netflow.log file?

[mjpworth]

the .netflow.log file attached as nfr2-mjp-test_netflow.log

[mjpworth]

.nextflow.log attached here as nfr2-mjp-test_nextflow.log thanks.

nfr2-mjp-test_nextflow.log

[m-bull]

Thanks for the log @mjpworth. Problem is here:

Caused by:
  Process requirement exceed available CPUs -- req: 16; avail: 1

Source block:
  new File("${params.outdir}/${params.prefix}.qc.csv").withWriter { writer ->
          for ( line in lines ) {
              writer.writeLine(line.join(','))
           }   
      }

Work dir:
  gs://covid-19-nysdohwc/scratch/89/6928e1811c8f8b97a5e3c057c8ca5e

Which I'd guess at being caused by wholesale setting cpus = 16 - probably related to this: https://github.com/connor-lab/ncov2019-artic-nf/issues/30.

If you add the following to the bottom of the file that you used to set cpus = 16 and memory = 24G, then you should be good:

process {
    withName: writeQCSummaryCSV {
        cpus = 1
    }
}

[mikek498]

I've tried running this process with cpus = 1 and it still fails.

The code appears to be nextflow native execution code without an output directive specified, could this be the issue?

`process writeQCSummaryCSV { tag { params.prefix } cpus 1 input: val lines

exec:
new File("${params.outdir}/${params.prefix}.qc.csv").withWriter { writer ->
    for ( line in lines ) {
        writer.writeLine(line.join(','))
     }   
}

}`

I've also noticed that the bucket referenced is missing a / in the gs://

`Apr-14 19:01:21.712 [Task submitter] INFO nextflow.Session - [19/8d25dc] Submitted process > ncovIllumina:sequenceAnalysis:writeQCSummaryCSV (outfile) Apr-14 19:01:21.719 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 23; name: ncovIllumina:sequenceAnalysis:writeQCSummaryCSV (outfile); status: COMPLETED; exit: -; error: java.io.FileNotFoundException: gs:/b iga-testing-life-sciences/output14/outfile.qc.csv (No such file or directory); workDir: gs://biga-testing-life-sciences/temp/19/8d25dc5a42bd568d4fc451c3c2796b] Apr-14 19:01:21.729 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'ncovIllumina:sequenceAnalysis:writeQCSummaryCSV (outfile)'

Caused by: gs:/biga-testing-life-sciences/output14/outfile.qc.csv (No such file or directory)

Source block: new File("${params.outdir}/${params.prefix}.qc.csv").withWriter { writer -> for ( line in lines ) { writer.writeLine(line.join(',')) }
}

Work dir: gs://biga-testing-life-sciences/temp/19/8d25dc5a42bd568d4fc451c3c2796b

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run Apr-14 19:01:21.800 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process ncovIllumina:sequenceAnalysis:writeQCSummaryCSV (outfile) failed `

[m-bull]

I think I've fixed this in https://github.com/connor-lab/ncov2019-artic-nf/commit/87568428fff980a86267ac384dd62cd351810e2d.

You're right about the output option, its related - I was writing outside the task work directory in this process, which is a no no. The commit fixes this and uses publishDir to push the result to GS etc.

[mikek498]

Thanks. The path references are now fixed however it now throws a new error, saying it cannot find the file. Looking at the bucket I cannot find the /temp/fd directory.

.nextflow.log

Apr-15 09:45:16.666 [Task submitter] INFO nextflow.Session - [fd/87a4e7] Submitted process > ncovIllumina:sequenceAnalysis:writeQCSummaryCSV (outfile) Apr-15 09:45:16.680 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 23; name: ncovIllumina:sequenceAnalysis:writeQCSummaryCSV (outfile); status: COMPLETED; exit: -; error: java.io.FileNotFoundException: /temp/fd/87a4e73870a0a61e0fcf70185ea6b9/outfile.qc.csv (No such file or directory); workDir: gs://biga-testing-life-sciences/temp/fd/87a4e73870a0a61e0fcf70185ea6b9] Apr-15 09:45:16.705 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'ncovIllumina:sequenceAnalysis:writeQCSummaryCSV (outfile)'

Caused by: /temp/fd/87a4e73870a0a61e0fcf70185ea6b9/outfile.qc.csv (No such file or directory)

Source block: new File("${task.workDir}/${params.prefix}.qc.csv").withWriter { writer -> for ( line in lines ) { writer.writeLine(line.join(',')) }
}

Work dir: gs://biga-testing-life-sciences/temp/fd/87a4e73870a0a61e0fcf70185ea6b9

[m-bull]

Thanks for testing @mikek498. Looks like I was using the wrong class to make this: https://github.com/nextflow-io/nextflow/issues/1185#issuecomment-505999978.

I've updated in https://github.com/connor-lab/ncov2019-artic-nf/commit/ca4c41de02aedfd9ddbf15c899607feb792bc5a4.

[mikek498]

I've just tried this updated code and it seems the file is still not being found in the gs bucket

Apr-15 10:47:28.877 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 23; name: nco vIllumina:sequenceAnalysis:writeQCSummaryCSV (outfile); status: COMPLETED; exit: -; error: java.nio.file.NoSuchFileExcep tion: /temp/de/3489326ba6b1b110ad6e2db6528c7c/outfile.qc.csv; workDir: gs://biga-testing-life-sciences/temp/de/3489326ba 6b1b110ad6e2db6528c7c] Apr-15 10:47:28.887 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'ncovIllumina:sequ enceAnalysis:writeQCSummaryCSV (outfile)'

Caused by: No such file: /temp/de/3489326ba6b1b110ad6e2db6528c7c/outfile.qc.csv

Source block: file("${task.workDir}/${params.prefix}.qc.csv").withWriter { writer -> for ( line in lines ) { writer.writeLine(line.join(',')) }
}

Work dir: gs://biga-testing-life-sciences/temp/de/3489326ba6b1b110ad6e2db6528c7c

[m-bull]

Looks like the failure is in the publishing of files. Final try to fix this in: https://github.com/connor-lab/ncov2019-artic-nf/commit/6feedcdae30e78236eaa03c876f1b80e4f11fa55.

[mikek498]

This update resolved the issue! Thanks.

[mjpworth]

@m-bull @mikek498 yes, works for me. thanks very much to both of you for your attention.