mc batch replication without S3 credentials

jhidalgo-lopez commented 8 months ago

Expected behavior

Considering BUCKET a public S3 bucket and the following replication.yaml:

replicate:
  apiVersion: v1
# SOURCE
  source:
    type: "s3"
    bucket: BUCKET
    prefix: ""
    endpoint: S3_ENDPOINT
    credentials:
      accessKey: ""
      secretKey: ""
# TARGET
  target:
    type: "minio"
    bucket: LOCAL_BUCKET

mc batch start ALIAS replication.yaml should work as mc mirror when setting up an mc alias such as mc alias set ALIAS S3_ENDPOINT without credentials: Replicating the public bucket using no S3 credentials.

Actual behavior

Task fails: mc: <ERROR> Unable to start job. Invalid arguments specified.

Steps to reproduce the behavior

Set up replication.yaml as stated with this:

S3_ ENDPOINT -> "https://s3-eu-west-1.amazonaws.com"
BUCKET -> "ngi-igenomes"
LOCAL_BUCKET -> any local bucket mc batch start ALIAS replication.yaml
mc --version

mc version RELEASE.2023-08-18T21-57-55Z (commit-id=acc25b8c5057e85dbca855f9d417f13de0603e60) Runtime: go1.19.12 linux/amd64 Copyright (c) 2015-2023 MinIO, Inc. License GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html

System information

Ubuntu 22.04 - amd64

vbob commented 2 months ago

Also facing this issue.

I want to sync from AWS open data https://registry.opendata.aws/ons-opendata-portal/

replication.yaml

replicate:
  apiVersion: v1
  source:
    type: s3 
    bucket: "ons-aws-prod-opendata"
    prefix: "" 
    endpoint: "https://ons-dl-prod-opendata.s3.amazonaws.com/"
    credentials:
      accessKey: "" 
      secretKey: "" 

  target:
    type: minio
    bucket: ons
    prefix: ""

mc batch start minio-local/ ./replication.yaml --debug --json

{
    "Code":"InvalidArgument",
    "Message":"Amazon S3 endpoint should be 's3.amazonaws.com'.",
    "Resource":"/minio/admin/v3/start-job",
    "RequestId":"17F14704E7EA3DB3",
    "HostId":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8"
}

Also tried with the s3.amazonaws.com and s3-sa-east-1.amazonaws.com with no luck:

"error": {
  "message": "Unable to start job",
  "cause": {
   "message": "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.",
   "error": {
    "Code": "PermanentRedirect",
    "Message": "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.",
    "BucketName": "",
    "Key": "",
    "RequestID": "17F147EA729BDE6F",
    "HostID": "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8",
    "Region": ""
   }
  },

vadmeste commented 2 months ago

@jhidalgo-lopez can you upgrade to the latest and try again ?

@vbob what is your MinIO version ?

minio / mc