Yesterday, I noticed a Nextflow run on AWS Batch failed even though all of the process steps completed. After looking at the log file, it says the pipeline terminated while publishing files, due to a java.nio.file.DirectoryNotEmptyException. I am using the mode: "copy" during publishing, and this particular process step failed to publish two small log files. This is using Nextflow version 23.10.0, launched manually from the CLI. This was not run with the -resume flag.
Expected behavior and actual behavior
Expected behavior: the process successfully copies the files to the destination publish directory, overwriting any files that are in the directory.
Actual behavior: the process failed to copy the files due to the destination path being a non-empty directory.
Steps to reproduce the problem
I have tried to reproduce this problem since, and it seems like this is on the rarer side. I've re-ran the same job with the same parameters and output folder destination several times both before and after this run, and there was no failure during publishing, even though the destination directory is not empty.
Program output
I've pasted part of the log fie from the failed run below. Let me know if more information is necessary.
May-21 19:21:58.521 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'PublishDir' minSize=10; maxSize=10; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
May-21 19:21:59.121 [AWSBatch-executor-17] DEBUG n.c.aws.batch.AwsBatchTaskHandler - [AWS BATCH] submitted > job=2a89ca72-a93f-481a-814a-4cc709a97699; work-dir=s3://go-development-nextflow-biofx-primary-ephemeral/240207_M07761_0036_000000000-GJ5RD-bcl2fastq-test-ET-148-no-experiment-name/52/d49e90ad1f1a68a069207c3c3d38bd
May-21 19:21:59.121 [AWSBatch-executor-17] INFO nextflow.Session - [52/d49e90] Submitted process > fastq_generation_rd_workflow:fastq_reporting_rd (1)
May-21 19:22:18.150 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 3; name: fastq_generation_rd_workflow:fastq_reporting_rd (1); status: COMPLETED; exit: 0; error: -; workDir: s3://go-development-nextflow-biofx-primary-ephemeral/240207_M07761_0036_000000000-GJ5RD-bcl2fastq-test-ET-148-no-experiment-name/52/d49e90ad1f1a68a069207c3c3d38bd]
May-21 19:22:18.253 [main] DEBUG nextflow.Session - Session await > all processes finished
May-21 19:22:18.273 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: awsbatch) - terminating tasks monitor poll loop
May-21 19:22:18.273 [main] DEBUG nextflow.Session - Session await > all barriers passed
May-21 19:22:23.275 [main] INFO nextflow.util.ThreadPoolHelper - Waiting for file transfers to complete (2 files)
May-21 19:23:23.276 [main] DEBUG nextflow.util.ThreadPoolHelper - Waiting for file transfers to complete (2 files)
May-21 19:23:47.593 [PublishDir-2] WARN nextflow.processor.PublishDir - Failed to publish file: s3://go-development-nextflow-biofx-primary-ephemeral/240207_M07761_0036_000000000-GJ5RD-bcl2fastq-test-ET-148-no-experiment-name/52/d49e90ad1f1a68a069207c3c3d38bd/240207_00000000-GJ5RD_CR23_003193_20240207_QC_JK_Testing_UPDATED; to: s3://go-development-nextflow-biofx-primary-ephemeral/240207_00000000-GJ5RD_CR23_003193_20240207_QC_JK_Testing_UPDATED [copy] -- See log file for details
java.nio.file.DirectoryNotEmptyException: the path: /go-development-nextflow-biofx-primary-ephemeral/240207_00000000-GJ5RD_CR23_003193_20240207_QC_JK_Testing_UPDATED is a directory and is not empty
at nextflow.cloud.aws.nio.S3FileSystemProvider.delete(S3FileSystemProvider.java:505)
at java.base/java.nio.file.Files.delete(Files.java:1152)
at nextflow.file.FileHelper$3.postVisitDirectory(FileHelper.groovy:999)
at nextflow.file.FileHelper$3.postVisitDirectory(FileHelper.groovy)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2828)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2882)
at nextflow.file.FileHelper.deleteDir0(FileHelper.groovy:991)
at nextflow.file.FileHelper.deletePath(FileHelper.groovy:983)
at nextflow.processor.PublishDir.processFile(PublishDir.groovy:382)
at nextflow.processor.PublishDir.safeProcessFile(PublishDir.groovy:353)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1254)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1030)
at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1036)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1019)
at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:97)
at nextflow.processor.PublishDir$_apply1_closure1.doCall(PublishDir.groovy:326)
at nextflow.processor.PublishDir$_apply1_closure1.call(PublishDir.groovy)
at groovy.lang.Closure.run(Closure.java:498)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
May-21 19:23:47.594 [PublishDir-2] DEBUG nextflow.Session - Session aborted -- Cause: the path: /go-development-nextflow-biofx-primary-ephemeral/240207_00000000-GJ5RD_CR23_003193_20240207_QC_JK_Testing_UPDATED is a directory and is not empty
May-21 19:23:58.941 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
May-21 19:23:58.943 [main] INFO nextflow.Nextflow - SW-007 Execution 74c55357-5dcf-4911-86dc-6b380cee9ae7 did not complete successfully.
May-21 19:23:58.946 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=3; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=7m 38s; failedDuratio
n=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=96; peakMemory=172 GB; ]
Bug report
Yesterday, I noticed a Nextflow run on AWS Batch failed even though all of the process steps completed. After looking at the log file, it says the pipeline terminated while publishing files, due to a
java.nio.file.DirectoryNotEmptyException. I am using themode: "copy"during publishing, and this particular process step failed to publish two small log files. This is using Nextflow version 23.10.0, launched manually from the CLI. This was not run with the-resumeflag.Expected behavior and actual behavior
Expected behavior: the process successfully copies the files to the destination publish directory, overwriting any files that are in the directory. Actual behavior: the process failed to copy the files due to the destination path being a non-empty directory.
Steps to reproduce the problem
I have tried to reproduce this problem since, and it seems like this is on the rarer side. I've re-ran the same job with the same parameters and output folder destination several times both before and after this run, and there was no failure during publishing, even though the destination directory is not empty.
Program output
I've pasted part of the log fie from the failed run below. Let me know if more information is necessary.
Environment