cannot stage in files from s3 compatible storage service with non-aws endpoint with newest Nextflow/nf-amazon version

katgorski commented 2 months ago

Bug report

Expected behavior and actual behavior

I am trying to stage in files stored on Wasabi (https://wasabi.com/), which is s3 compatible. The code below fails with Nextflow version 23.10.1, but runs successfully with the command NXF_VER=23.04.5 nextflow run main.nf -c nextflow.config (which uses nf-amazon@1.16.2).

Steps to reproduce the problem

main.nf

workflow {
    WASABI_STAGE_WASABI_PUB(params.wasabi_path)
}

process WASABI_STAGE_WASABI_PUB {

    publishDir "${params.wasabi_publish_dir}", saveAs: { filename -> "${new Date().format('yyyyMMddHHmmss')}_${task.executor}_${filename}" }

    input:
    path(wasabi_path)

    output:
    path("wasabi_to_wasabi.txt")

    script:
    """
    ls -al . > wasabi_to_wasabi.txt
    """

}

nextflow.config

params {
    wasabi_path = 's3://s3-bucket/s3-filepath/s3-file'
    wasabi_publish_dir = 's3://s3-bucket/nextflow'
}

aws {
    accessKey = '---'
    secretKey = '---'
    client {
        endpoint = 'https://s3.us-east-2.wasabisys.com'
        s3PathStyleAccess = true
    }
}

Command: nextflow run main.nf -c nextflow.config

Program output

N E X T F L O W  ~  version 23.10.1
Launching `main.nf` [silly_poitras] DSL2 - revision: 4556fd0279
[-        ] process > WASABI_STAGE_WASABI_PUB -
WARN: Unable to get file attributes file: s3://s3-bucket/s3-filepath/s3-file -- Cause: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exi
[-        ] process > WASABI_STAGE_WASABI_PUB -
WARN: Unable to get file attributes file: s3://s3-bucket/s3-filepath/s3-file -- Cause: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: JC8X87T8A3BT3Y8E; S3 Extended Request ID: sajJzdE1PhC2JTBwH3W7Qc4g8GfwKjPIlgGv73eeHe8IS2m7zf+m5eiGJxRa8CpYTZU2l61t61A=; Proxy
[-        ] process > WASABI_STAGE_WASABI_PUB -
WARN: Unable to get file attributes file: s3://s3-bucket/s3-filepath/s3-file -- Cause: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: JC8X87T8A3BT3Y8E; S3 Extended Request ID: sajJzdE1PhC2JTBwH3W7Qc4g8GfwKjPIlgGv73eeHe8IS2m7zf+m5eiGJxRa8CpYTZU2l61t61A=; Proxy: null), S3 Extended Request ID: sajJzdE1PhC2JTBwH3W7Qc4g8GfwKjPIlgGv73eeHe8IS2m7zf+m5eiGJxRa8CpYTZU2l61t61A=
ERROR ~ Error executing process > 'WASABI_STAGE_WASABI_PUB'

Caused by:
  The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: JC8H0E7X9W2WAEM0; S3 Extended Request ID: pNZ62Q25W+xIKP5O4iyld4b3yu9EMd64bjRGo3qJRoRlrqmlgqljuIQQtXlvI8xI268LJkQAmY4=; Proxy: null)

 -- Check '.nextflow.log' file for details

nextflow log: nextflow.log.23.10.1.txt

Environment

Nextflow version: 23.10.1 (nf-amazon@2.1.4)
Java version: 17.0.10
Operating system: macOS Sonoma 14.4.1 (however, this also occurs on Ubuntu 22.04.4 LTS)
Bash version: zsh 5.9 (x86_64-apple-darwin23.0) (also occurs on GNU bash 5.1.16)

Additional context

Again, this works as expected with NXF_VER=23.04.5 nextflow run main.nf -c nextflow.config (Java, OS, zsh version the same). Logs for that execution: nextflow.log.23.04.5.txt

bentsherman commented 2 months ago

This error usually means some part of your credentials are missing. Do you also have a session token in addition to access key and secret key?

katgorski commented 2 months ago

This error usually means some part of your credentials are missing. Do you also have a session token in addition to access key and secret key?

I do not. I drove myself crazy trying to determine if the credentials were OK or not, but passing in just the keys in 23.04 was fine (same credentials across the 23.04 and 23.10 executions). I know nf-amazon was overhauled between the two releases, but I'm not familiar with the interfaces used for interacting with aws. Theoretically this could be something specific to my endpoint, but I'm not sure.

nextflow-io / nextflow