seqeralabs / nf-tower

Nextflow Tower system
https://tower.nf
Mozilla Public License 2.0
143 stars 51 forks source link

publishDir fails to publish files to s3 when >5gb #362

Open qscacheri opened 1 year ago

qscacheri commented 1 year ago

When the result of a process is larger than 5gb, the upload to s3 fails with the error:

The specified copy source is larger than the maximum allowable size for a copy source: 5368709120 (Service: Amazon S3; Status Code: 400; Error Code: InvalidRequest

Minimal reproducible example:

nextflow.enable.dsl=2

process createFile {
    publishDir "s3://some-bucket", mode: 'copy', overwrite: true, failOnError: true
    output:
        path("*.gz")
    shell:
    '''
    truncate -s 10G test.tar.gz
    '''
}

workflow {
    createFile()
    createFile.out.view { "File: $it" }
}