hangs because it tries to stage NO_REG_BED which doesn't exist

[esrice]

Operating System

Ubuntu 22.04

Other Linux

No response

Workflow Version

v0.5.3-g8b9748b

Workflow Execution

Command line

EPI2ME Version

No response

CLI command run

project_dir="gs://my-bucket-name/project-name/"
nextflow run epi2me-labs/wf-clone-validation \
    -work-dir $project_dir/nxf-work \
    --fastq $project_dir/example_data \
    --host_reference $project_dir/host.fa \
    --out_dir $project_dir/example_runs/231122

Workflow Execution - CLI Execution Profile

custom

What happened?

I am running the pipeline using nextflow's google cloud batch mode. I am not specifying a regions bed file, so it looks like the pipeline sets the bed file to a placeholder string NO_REG_BED:

https://github.com/epi2me-labs/wf-clone-validation/blob/8b9748bc00aac4f531778d7d6d2f9278f26e4201/main.nf#L691

This works when using an executor that does not need to do any staging, because checkIfExists is set to false and then the process filterHostReads checks to see whether it actually exists in the bash script:

https://github.com/epi2me-labs/wf-clone-validation/blob/8b9748bc00aac4f531778d7d6d2f9278f26e4201/main.nf#L69

However, when using an executor like google cloud that does staging, nextflow tries to copy this nonexistent file and then gets confused (see log).

To get around this issue, I just ran

touch NO_REG_BED

before the nextflow command. See this issue for discussion of various hacky ways to get around this that don't break cloud executors: https://github.com/nextflow-io/nextflow/issues/1694

Relevant log output

Nov-22 16:11:25.701 [Actor Thread 18] DEBUG nextflow.util.CacheHelper - Unable to get file attributes file: /home/edward/NO_REG_BED -- Cause: java.nio.file.NoSuchFileException: /home/edward/NO_REG_BED
Nov-22 16:11:25.799 [FileTransfer-2] DEBUG nextflow.file.FilePorter - Copying foreign file /home/edward/NO_REG_BED to work dir: gs://esr-bucket1/plasmids-project/nxf-work/stage-5c849075-a59a-492a-80a2-2adb7501750b/ff/42fd5bc2599d7c63de84cfae95eb54/NO_REG_BED
  error [nextflow.exception.ProcessStageException]: Can't stage file file:///home/edward/NO_REG_BED -- file does not exist
Nov-22 16:11:25.807 [Actor Thread 18] INFO  nextflow.processor.TaskProcessor - [null] NOTE: Can't stage file file:///home/edward/NO_REG_BED -- file does not exist -- Error is ignored
Nov-22 16:15:05.537 [Actor Thread 20] DEBUG nextflow.util.CacheHelper - Unable to get file attributes file: /home/edward/NO_REG_BED -- Cause: java.nio.file.NoSuchFileException: /home/edward/NO_REG_BED
  error [nextflow.exception.ProcessStageException]: Can't stage file file:///home/edward/NO_REG_BED -- file does not exist
Nov-22 16:15:05.542 [Actor Thread 20] INFO  nextflow.processor.TaskProcessor - [null] NOTE: Can't stage file file:///home/edward/NO_REG_BED -- file does not exist -- Error is ignored
Nov-22 16:18:55.583 [Actor Thread 19] DEBUG nextflow.util.CacheHelper - Unable to get file attributes file: /home/edward/NO_REG_BED -- Cause: java.nio.file.NoSuchFileException: /home/edward/NO_REG_BED
  error [nextflow.exception.ProcessStageException]: Can't stage file file:///home/edward/NO_REG_BED -- file does not exist
Nov-22 16:18:55.587 [Actor Thread 19] INFO  nextflow.processor.TaskProcessor - [null] NOTE: Can't stage file file:///home/edward/NO_REG_BED -- file does not exist -- Error is ignored
Nov-22 16:22:35.514 [Actor Thread 24] DEBUG nextflow.util.CacheHelper - Unable to get file attributes file: /home/edward/NO_REG_BED -- Cause: java.nio.file.NoSuchFileException: /home/edward/NO_REG_BED
  error [nextflow.exception.ProcessStageException]: Can't stage file file:///home/edward/NO_REG_BED -- file does not exist
Nov-22 16:22:35.515 [Actor Thread 24] INFO  nextflow.processor.TaskProcessor - [null] NOTE: Can't stage file file:///home/edward/NO_REG_BED -- file does not exist -- Error is ignored

Application activity log entry

No response

[sarahjeeeze]

Hi, thanks for letting us know, we have a fix incoming for this shortly.

[sarahjeeeze]

Hi, a fix for this should now be in the latest version.

[sarahjeeeze]

Closing as this is now fixed.