Nextflow creates object ending with slash during when using publishDir

[adriannavarrobetrian]

Bug report

When using the publishDir directive to send outputs to an s3 location, it looks like Nextflow creates a zero-sized object with a key ending in a slash at the publishDir location. While technically allowed by S3, this creates issues when performing operations on the resulting publishDir location (like recursing over objects or counting the number of objects under a prefix). It will also keep empty prefixes around; the objects themselves cannot be seen in the console, and if you try to copy the prefix using the AWS CLI naively, it fails.

Steps to reproduce the problem

Minimal example to replicate:

process example {
    publishDir "s3://my-bucket/buy-why-nextflow/test"

    input:
    val sample

    output:
    path "*fastq.gz"

    script:
    """
    touch ${sample}.fastq.gz
    """
}

workflow {
    example(Channel.of("SAMP1", "SAMP2"))
}

$ aws s3 ls --recursive s3://my-bucket/test/
2024-06-13 15:15:04          0 test/
2024-06-13 15:15:04          0 test/SAMP1.fastq.gz
2024-06-13 15:15:03          0 test/SAMP2.fastq.gz

Program output

$ aws s3 cp s3://my-bucket/test/ ./
download failed: s3://my-bucket/test/ to ./ [Errno 21] Is a directory: '/some_local_dir/.5beAaC30' -> '/some_local_dir/'

Environment

• Nextflow version: 23.10.1

[pditommaso]

I'm not understanding what's supposed to be prefix in your example.

I've used this process definition

process example {
    publishDir "s3://nextflow-ci/buy-why-nextflow"

    input:
    val sample

    output:
    path "*fastq.gz"

    script:
    """
    touch ${sample}.fastq.gz
    """
}

I'm getting this result that's perfectly fine

2024-06-18 18:16:52          0 
2024-06-18 18:16:52          0 SAMP1.fastq.gz
2024-06-18 18:16:52          0 SAMP2.fastq.gz

[adriannavarrobetrian]

Sorry, I copied the example wrong. It's a folder, I updated it to test.

[pditommaso]

It's essentially the same, I don't see why it should not work