Downloading an artifact from a private s3 fails when directly specifying IAM keys

Nomad version

Nomad v1.7.6 BuildDate 2024-03-12T07:27:36Z Revision 594fedbfbc4f0e532b65e8a69b28ff9403eb822e

Operating system and Environment details

Linux 6.5.0-18-generic 18~22.04.1-Ubuntu x86_64

Issue

The documentation for the artifact states that this is one of the allowed formats when downloading from s3:

artifact {
  source = "s3://my-bucket-example.s3-us-west-2.amazonaws.com/my_app.tar.gz"
}

This works correctly when the IAM credentials are read from the host, however when I specify those credentials directly, that bucket format doesn't work.

Reproduction steps

Trying to fetch an artifact like this:

artifact {
  source = "s3://my-bucket-example.s3-us-west-2.amazonaws.com/my_app.tar.gz"
  options {
    aws_access_key_id     = "<id>"
    aws_access_key_secret = "<secret>"
  }
}

The error is similar to:

failed to download artifact "s3://my-bucket-example.s3-us-west-2.amazonaws.com/my_app.tar.gz": getter subprocess failed: exit status 1: failed to download artifact: AccessDenied: Access Denied status code: 403, request id: XYZ, host id: w9ypewbKeo[......]4+Nep=

By just removing s3:// from the source, the object can be fetched now:

artifact {
  source = "my-bucket-example.s3-us-west-2.amazonaws.com/my_app.tar.gz"
  options {
    aws_access_key_id     = "<id>"
    aws_access_key_secret = "<secret>"
  }
}

This works both when explicitly specifying the credentials and when they are fetched from the credentials file. Haven't tested this with ec2 instance profiles.

hashicorp / go-getter