search

nextflow-io / nextflow

A DSL for data-driven computational pipelines
http://nextflow.io
Apache License 2.0
2.76k stars 629 forks source link

WARN: Failed to publish file (S3, move only) #2267

Closed bsiranosian closed 2 years ago

bsiranosian commented 3 years ago

Bug report

Running a pipeline with aws batch executor, and all files on S3. PublishDir fails when mode is 'move' but not 'copy'. Adding overwirite: 'true' doesn't help. When mode is 'copy' everything works as expected. I'd rather not keep additional copies of the large sequencing files on S3 if I don't have to.

Expected behavior and actual behavior

Expected: Files are moved from workdir into destination on S3 Actual: Files are not moved and remain in workdir

Steps to reproduce the problem

Can provide minimal example later if necessary. An example rule:

params.outdir_fastqc_pre = "s3://MY_BUCKET/sequence_files_processing/fastqc_pre/$SOURCE_NAME/$SOURCE_ID/"
process fastqc_pre {
    tag "$SOURCE_ID"
    cpus 2
    memory '4 GB'
    publishDir params.outdir_fastqc_pre , mode: 'move', overwrite: 'true'
    queue 'fastqc_bioinformatics    '
    container "biocontainers/fastqc:v0.11.9_cv8"

    input:
    path(reads) from read_pairs_ch1
    output:
    path("*fastqc*")
    script:
    """
    echo $SOURCE_ID    
    echo $SOURCE_NAME
    fastqc -t ${task.cpus} $reads
    """

Program output

S3 paths obscured here. I ran this example with -resume because the files were generated in a previous run. But the behavior is the same if starting from scratch

$ nextflow run ~/projects/data-processing/src/nextflow/preprocessing.nf -bucket-dir s3://MY_BUCKET/bsiranos -c ~/projects/data-processing/src/nextflow/preprocessing.config -resume
N E X T F L O W  ~  version 21.04.3
Launching `/home/bsiranos/projects/data-processing/src/nextflow/preprocessing.nf` [big_kare] - revision: d0b54f29d7
[fd/ce0d83] process > fastqc_pre (SRR2889328_short)  [100%] 1 of 1, cached: 1
[47/9a4354] process > trim_galore (SRR2889328_short) [100%] 1 of 1, cached: 1 ✔
[fd/ce0d83] process > fastqc_pre (SRR2889328_short)  [100%] 1 of 1, cached: 1 ✔
[47/9a4354] process > trim_galore (SRR2889328_short) [100%] 1 of 1, cached: 1 ✔
[-        ] process > fastqc_post                    -
WARN: Failed to publish file: /MY_BUCKET/bsiranos/47/9a4354abc938439afd7bc200170f43/SRR2889328_short_trimmed.fq.gz; to: /MY_BUCKET/sequence_files_processing/processed/SRA/SRR2889328_short/SRR2889328_short_trimmed.fq.gz [move] -- See log file for de[fd/ce0d83] process > fastqc_pre (SRR2889328_short)  [100%] 1 of 1, cached: 1 ✔
[47/9a4354] process > trim_galore (SRR2889328_short) [100%] 1 of 1, cached: 1 ✔
[2d/9a20d7] process > fastqc_post (SRR2889328_short) [100%] 1 of 1, cached: 1 ✔
WARN: Failed to publish file: /MY_BUCKET/bsiranos/47/9a4354abc938439afd7bc200170f43/SRR2889328_short_trimmed.fq.gz; to: /MY_BUCKET/sequence_files_processing/processed/SRA/SRR2889328_short/SRR2889328_short_trimmed.fq.gz [move] -- See log file for details
WARN: Failed to publish file: /MY_BUCKET/bsiranos/fd/ce0d8373cfd059303fe3621b65c356/SRR2889328_short_fastqc.zip; to: /MY_BUCKET/sequence_files_processing/fastqc_pre/SRA/SRR2889328_short/SRR2889328_short_fastqc.zip [move] -- See log file for details
WARN: Failed to publish file: /MY_BUCKET/bsiranos/fd/ce0d8373cfd059303fe3621b65c356/SRR2889328_short_fastqc.html; to: /bioin[fd/ce0d83] process > fastqc_pre (SRR2889328_short)  [100%] 1 of 1, cached: 1 ✔
[47/9a4354] process > trim_galore (SRR2889328_short) [100%] 1 of 1, cached: 1 ✔
[2d/9a20d7] process > fastqc_post (SRR2889328_short) [100%] 1 of 1, cached: 1 ✔
WARN: Failed to publish file: /MY_BUCKET/bsiranos/47/9a4354abc938439afd7bc200170f43/SRR2889328_short_trimmed.fq.gz; to: /MY_BUCKET/sequence_files_processing/processed/SRA/SRR2889328_short/SRR2889328_short_trimmed.fq.gz [move] -- See log file for details
WARN: Failed to publish file: /MY_BUCKET/bsiranos/fd/ce0d8373cfd059303fe3621b65c356/SRR2889328_short_fastqc.zip; to: /MY_BUCKET/sequence_files_processing/fastqc_pre/SRA/SRR2889328_short/SRR2889328_short_fastqc.zip [move] -- See log file for details
WARN: Failed to publish file: /MY_BUCKET/bsiranos/fd/ce0d8373cfd059303fe3621b65c356/SRR2889328_short_fastqc.html; to: /MY_BUCKET/sequence_files_processing/fastqc_pre/SRA/SRR2889328_short/SRR2889328_short_fastqc.html [move] -- See log file for details
WARN: Failed to publish file: /MY_BUCKET/bsiranos/2d/9a20d74cbbf383fba60d44459176dd/SRR2889328_short_trimmed_fastqc.zip; to: /MY_BUCKET/sequence_files_processing/fastqc_post/SRA/SRR2889328_short/SRR2889328_short_trimmed_fastqc.zip [move] -- See log file for details
WARN: Failed to publish file: /MY_BUCKET/bsiranos/2d/9a20d74cbbf383fba60d44459176dd/SRR2889328_short_trimmed_fastqc.html; to: /MY_BUCKET/sequence_files_processing/fastqc_post/SRA/SRR2889328_short/SRR2889328_short_trimmed_fastqc.html [move] -- See log file for details

Environment

Additional context

Nextflow log:

Aug-18 15:24:44.897 [main] DEBUG nextflow.cli.Launcher - $> nextflow run /home/bsiranos/projects/data-processing/src/nextflow/preprocessing.nf -bucket-dir 's3://MY_BUCKET/bsiranos' -c /home/bsiranos/projects/data-processing/src/nextflow/preprocessing.config -resume
Aug-18 15:24:45.018 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 21.04.3
Aug-18 15:24:45.040 [main] INFO  nextflow.cli.CmdRun - Launching `/home/bsiranos/projects/data-processing/src/nextflow/preprocessing.nf` [big_kare] - revision: d0b54f29d7
Aug-18 15:24:45.058 [main] DEBUG nextflow.config.ConfigBuilder - User config file: /home/bsiranos/projects/data-processing/src/nextflow/preprocessing.config
Aug-18 15:24:45.059 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /home/bsiranos/projects/data-processing/src/nextflow/preprocessing.config
Aug-18 15:24:45.077 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `standard`
Aug-18 15:24:45.591 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; plugins-dir=/home/bsiranos/.nextflow/plugins
Aug-18 15:24:45.593 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[nf-amazon@1.0.5]
Aug-18 15:24:45.597 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins local root: .nextflow/plr/87881143a005e0d03979c7dc22790185
Aug-18 15:24:45.603 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Aug-18 15:24:45.604 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Aug-18 15:24:45.607 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Aug-18 15:24:45.621 [main] INFO  org.pf4j.AbstractPluginManager - Plugin 'nf-amazon@1.0.5' resolved
Aug-18 15:24:45.627 [main] DEBUG nextflow.plugin.PluginUpdater - Starting plugin nf-amazon version: 1.0.5
Aug-18 15:24:45.627 [main] INFO  org.pf4j.AbstractPluginManager - Start plugin 'nf-amazon@1.0.5'
Aug-18 15:24:45.640 [main] DEBUG nextflow.plugin.BasePlugin - Plugin started nf-amazon@1.0.5
Aug-18 15:24:45.655 [main] DEBUG nextflow.file.FileHelper - > Added 'S3FileSystemProvider' to list of installed providers [s3]
Aug-18 15:24:45.699 [main] DEBUG nextflow.Session - Session uuid: 0ae0be03-cda9-4d5a-9793-040fdec51890
Aug-18 15:24:45.699 [main] DEBUG nextflow.Session - Run name: big_kare
Aug-18 15:24:45.700 [main] DEBUG nextflow.Session - Executor pool size: 12
Aug-18 15:24:45.731 [main] DEBUG nextflow.cli.CmdRun - 
  Version: 21.04.3 build 5560
  Created: 21-07-2021 15:09 UTC (08:09 PDT)
  System: Linux 5.4.0-80-generic
  Runtime: Groovy 3.0.7 on OpenJDK 64-Bit Server VM 11.0.9.1-internal+0-adhoc..src
  Encoding: UTF-8 (UTF-8)
  Process: 405409@xpsxps [127.0.1.1]
  CPUs: 12 - Mem: 15.3 GB (497.7 MB) - Swap: 2 GB (1.5 GB)
Aug-18 15:24:45.765 [main] DEBUG nextflow.Session - Work-dir: /home/bsiranos/nf_test/work [ext2/ext3]
Aug-18 15:24:45.771 [main] DEBUG nextflow.file.FileHelper - Creating a file system instance for provider: S3FileSystemProvider
Aug-18 15:24:45.777 [main] DEBUG nextflow.Global - Using AWS credentials defined by environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
Aug-18 15:24:45.777 [main] DEBUG nextflow.file.FileHelper - AWS S3 config details: {secret_key=/etntI.., region=us-west-2, access_key=AKIA2S..}
Aug-18 15:24:46.167 [main] DEBUG nextflow.Session - Bucket-dir: s3://MY_BUCKET/bsiranos
Aug-18 15:24:46.167 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /home/bsiranos/projects/data-processing/src/nextflow/bin
Aug-18 15:24:46.195 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[AwsBatchExecutor]
Aug-18 15:24:46.206 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Aug-18 15:24:46.316 [main] DEBUG nextflow.Session - Session start invoked
Aug-18 15:24:46.725 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Aug-18 15:24:46.736 [main] DEBUG nextflow.Session - Workflow process names [dsl1]: fastqc_post, fastqc_pre, trim_galore
Aug-18 15:24:46.767 [PathVisitor-1] DEBUG nextflow.file.PathVisitor - files for syntax: glob; folder: /MY_BUCKET/sequence_files/SRA/SRR2889328_short/; pattern: SRR2889328_short*.[fq|fastq]*; options: [:]
Aug-18 15:24:46.769 [PathVisitor-1] DEBUG nextflow.file.FileHelper - Path matcher not defined by 'S3FileSystem' file system -- using default default strategy
Aug-18 15:24:46.828 [PathVisitor-2] DEBUG nextflow.file.PathVisitor - files for syntax: glob; folder: /MY_BUCKET/sequence_files/SRA/SRR2889328_short/; pattern: SRR2889328_short*.[fq|fastq]*; options: [:]
Aug-18 15:24:46.828 [PathVisitor-2] DEBUG nextflow.file.FileHelper - Path matcher not defined by 'S3FileSystem' file system -- using default default strategy
Aug-18 15:24:46.902 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: awsbatch
Aug-18 15:24:46.902 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'awsbatch'
Aug-18 15:24:46.904 [main] DEBUG nextflow.executor.Executor - [warm up] executor > awsbatch
Aug-18 15:24:46.915 [main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-executor, limiter:RateLimiter[stableRate=50.0qps], poolSize:60, maxPoolSize:60, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
Aug-18 15:24:46.920 [main] DEBUG nextflow.util.ThrottlingExecutor - Creating throttling executor with opts: nextflow.util.ThrottlingExecutor$Options(poolName:AWSBatch-reaper, limiter:RateLimiter[stableRate=50.0qps], poolSize:60, maxPoolSize:60, queueSize:5000, maxRetries:10, keepAlive:1m, autoThrottle:true, errorBurstDelay:1s, rampUpInterval:100, rampUpFactor:1.2, rampUpMaxRate:1.7976931348623157E308, backOffFactor:2.0, backOffMinRate:0.0166666667, retryDelay:1s)
Aug-18 15:24:46.921 [main] DEBUG n.cloud.aws.batch.AwsBatchExecutor - Creating parallel monitor for executor 'awsbatch' > pollInterval=10s; dumpInterval=5m
Aug-18 15:24:46.943 [main] DEBUG nextflow.Global - Using AWS credentials defined by environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
Aug-18 15:24:47.224 [main] DEBUG n.cloud.aws.batch.AwsBatchExecutor - [AWS BATCH] Executor options=AwsOptions(cliPath:/home/ec2-user/miniconda/bin/aws, storageClass:null, storageEncryption:null, remoteBinDir:null, region:us-west-2, maxParallelTransfers:16, maxTransferAttempts:1, delayBetweenAttempts:10s, fetchInstanceType:false, jobRole:null, volumes:[], awsCli:/home/ec2-user/miniconda/bin/aws --region us-west-2)
Aug-18 15:24:47.293 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: awsbatch
Aug-18 15:24:47.293 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'awsbatch'
Aug-18 15:24:47.302 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: awsbatch
Aug-18 15:24:47.303 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'awsbatch'
Aug-18 15:24:47.306 [main] DEBUG nextflow.script.ScriptRunner - > Await termination 
Aug-18 15:24:47.306 [main] DEBUG nextflow.Session - Session await
Aug-18 15:24:48.079 [Actor Thread 1] DEBUG nextflow.file.FileHelper - Path matcher not defined by 'S3FileSystem' file system -- using default default strategy
Aug-18 15:24:48.079 [Actor Thread 4] DEBUG nextflow.file.FileHelper - Path matcher not defined by 'S3FileSystem' file system -- using default default strategy
Aug-18 15:24:48.208 [Actor Thread 4] INFO  nextflow.processor.TaskProcessor - [fd/ce0d83] Cached process > fastqc_pre (SRR2889328_short)
Aug-18 15:24:48.208 [Actor Thread 1] INFO  nextflow.processor.TaskProcessor - [47/9a4354] Cached process > trim_galore (SRR2889328_short)
Aug-18 15:24:48.290 [Actor Thread 1] WARN  com.amazonaws.util.Base64 - JAXB is unavailable. Will fallback to SDK implementation which may be less performant
Aug-18 15:24:48.293 [Actor Thread 1] DEBUG n.util.BlockingThreadExecutorFactory - Thread pool name=FileTransfer; maxThreads=36; maxQueueSize=108; keepAlive=1m
Aug-18 15:24:48.300 [FileTransfer-thread-1] WARN  nextflow.processor.PublishDir - Failed to publish file: /MY_BUCKET/bsiranos/47/9a4354abc938439afd7bc200170f43/SRR2889328_short_trimmed.fq.gz; to: /MY_BUCKET/sequence_files_processing/processed/SRA/SRR2889328_short/SRR2889328_short_trimmed.fq.gz [move] -- See log file for details
java.lang.UnsupportedOperationException: null
    at com.upplication.s3fs.S3FileSystemProvider.move(S3FileSystemProvider.java:580)
    at nextflow.file.FileHelper.movePath(FileHelper.groovy:860)
    at nextflow.processor.PublishDir.processFileImpl(PublishDir.groovy:300)
    at nextflow.processor.PublishDir.processFile(PublishDir.groovy:274)
    at nextflow.processor.PublishDir.safeProcessFile(PublishDir.groovy:260)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1268)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
    at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1029)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1012)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:101)
    at nextflow.processor.PublishDir$_apply1_closure1.doCall(PublishDir.groovy:233)
    at nextflow.processor.PublishDir$_apply1_closure1.call(PublishDir.groovy)
    at groovy.lang.Closure.run(Closure.java:493)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Aug-18 15:24:48.439 [FileTransfer-thread-2] WARN  nextflow.processor.PublishDir - Failed to publish file: /MY_BUCKET/bsiranos/fd/ce0d8373cfd059303fe3621b65c356/SRR2889328_short_fastqc.zip; to: /MY_BUCKET/sequence_files_processing/fastqc_pre/SRA/SRR2889328_short/SRR2889328_short_fastqc.zip [move] -- See log file for details
java.lang.UnsupportedOperationException: null
    at com.upplication.s3fs.S3FileSystemProvider.move(S3FileSystemProvider.java:580)
    at nextflow.file.FileHelper.movePath(FileHelper.groovy:860)
    at nextflow.processor.PublishDir.processFileImpl(PublishDir.groovy:300)
    at nextflow.processor.PublishDir.processFile(PublishDir.groovy:274)
    at nextflow.processor.PublishDir.safeProcessFile(PublishDir.groovy:260)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1268)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
    at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1029)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1012)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:101)
    at nextflow.processor.PublishDir$_apply1_closure1.doCall(PublishDir.groovy:233)
    at nextflow.processor.PublishDir$_apply1_closure1.call(PublishDir.groovy)
    at groovy.lang.Closure.run(Closure.java:493)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Aug-18 15:24:48.439 [Actor Thread 8] DEBUG nextflow.file.FileHelper - Path matcher not defined by 'S3FileSystem' file system -- using default default strategy
Aug-18 15:24:48.439 [FileTransfer-thread-3] WARN  nextflow.processor.PublishDir - Failed to publish file: /MY_BUCKET/bsiranos/fd/ce0d8373cfd059303fe3621b65c356/SRR2889328_short_fastqc.html; to: /MY_BUCKET/sequence_files_processing/fastqc_pre/SRA/SRR2889328_short/SRR2889328_short_fastqc.html [move] -- See log file for details
java.lang.UnsupportedOperationException: null
    at com.upplication.s3fs.S3FileSystemProvider.move(S3FileSystemProvider.java:580)
    at nextflow.file.FileHelper.movePath(FileHelper.groovy:860)
    at nextflow.processor.PublishDir.processFileImpl(PublishDir.groovy:300)
    at nextflow.processor.PublishDir.processFile(PublishDir.groovy:274)
    at nextflow.processor.PublishDir.safeProcessFile(PublishDir.groovy:260)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1268)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
    at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1029)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1012)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:101)
    at nextflow.processor.PublishDir$_apply1_closure1.doCall(PublishDir.groovy:233)
    at nextflow.processor.PublishDir$_apply1_closure1.call(PublishDir.groovy)
    at groovy.lang.Closure.run(Closure.java:493)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Aug-18 15:24:48.635 [Actor Thread 8] INFO  nextflow.processor.TaskProcessor - [2d/9a20d7] Cached process > fastqc_post (SRR2889328_short)
Aug-18 15:24:48.694 [FileTransfer-thread-4] WARN  nextflow.processor.PublishDir - Failed to publish file: /MY_BUCKET/bsiranos/2d/9a20d74cbbf383fba60d44459176dd/SRR2889328_short_trimmed_fastqc.zip; to: /MY_BUCKET/sequence_files_processing/fastqc_post/SRA/SRR2889328_short/SRR2889328_short_trimmed_fastqc.zip [move] -- See log file for details
java.lang.UnsupportedOperationException: null
    at com.upplication.s3fs.S3FileSystemProvider.move(S3FileSystemProvider.java:580)
    at nextflow.file.FileHelper.movePath(FileHelper.groovy:860)
    at nextflow.processor.PublishDir.processFileImpl(PublishDir.groovy:300)
    at nextflow.processor.PublishDir.processFile(PublishDir.groovy:274)
    at nextflow.processor.PublishDir.safeProcessFile(PublishDir.groovy:260)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1268)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
    at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1029)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1012)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:101)
    at nextflow.processor.PublishDir$_apply1_closure1.doCall(PublishDir.groovy:233)
    at nextflow.processor.PublishDir$_apply1_closure1.call(PublishDir.groovy)
    at groovy.lang.Closure.run(Closure.java:493)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Aug-18 15:24:48.694 [FileTransfer-thread-5] WARN  nextflow.processor.PublishDir - Failed to publish file: /MY_BUCKET/bsiranos/2d/9a20d74cbbf383fba60d44459176dd/SRR2889328_short_trimmed_fastqc.html; to: /MY_BUCKET/sequence_files_processing/fastqc_post/SRA/SRR2889328_short/SRR2889328_short_trimmed_fastqc.html [move] -- See log file for details
java.lang.UnsupportedOperationException: null
    at com.upplication.s3fs.S3FileSystemProvider.move(S3FileSystemProvider.java:580)
    at nextflow.file.FileHelper.movePath(FileHelper.groovy:860)
    at nextflow.processor.PublishDir.processFileImpl(PublishDir.groovy:300)
    at nextflow.processor.PublishDir.processFile(PublishDir.groovy:274)
    at nextflow.processor.PublishDir.safeProcessFile(PublishDir.groovy:260)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1268)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
    at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1029)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1012)
    at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:101)
    at nextflow.processor.PublishDir$_apply1_closure1.doCall(PublishDir.groovy:233)
    at nextflow.processor.PublishDir$_apply1_closure1.call(PublishDir.groovy)
    at groovy.lang.Closure.run(Closure.java:493)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Aug-18 15:24:48.695 [main] DEBUG nextflow.Session - Session await > all process finished
Aug-18 15:24:56.936 [main] DEBUG nextflow.Session - Session await > all barriers passed
Aug-18 15:24:56.954 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=0; ignoredCount=0; cachedCount=3; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=0ms; cachedDuration=1m 52s;loadCpus=0; loadMemory=0; peakRunning=0; peakCpus=0; peakMemory=0; ]
Aug-18 15:24:56.994 [main] DEBUG nextflow.CacheDB - Closing CacheDB done
Aug-18 15:24:56.994 [main] INFO  org.pf4j.AbstractPluginManager - Stop plugin 'nf-amazon@1.0.5'
Aug-18 15:24:56.994 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-amazon
Aug-18 15:24:57.014 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

pditommaso commented 1 year ago

Hi @bsiranosian, 21.04.3 is a two years old version. Please try updating to latest 23.04.2

bsiranosian commented 1 year ago

Well, I think that was the current version at the time of issue submission :) I'm not sure if this is still an issue in 23.04

pditommaso commented 1 year ago