mc batch replication without S3 credentials

jhidalgo-lopez commented 9 months ago

Expected behavior

Considering BUCKET a public S3 bucket and the following replication.yaml:

replicate:
  apiVersion: v1
# SOURCE
  source:
    type: "s3"
    bucket: BUCKET
    prefix: ""
    endpoint: S3_ENDPOINT
    credentials:
      accessKey: ""
      secretKey: ""
# TARGET
  target:
    type: "minio"
    bucket: LOCAL_BUCKET

mc batch start ALIAS replication.yaml should work as mc mirror when setting up an mc alias such as mc alias set ALIAS S3_ENDPOINT without credentials: Replicating the public bucket using no S3 credentials.

Actual behavior

Task fails: mc: <ERROR> Unable to start job. Invalid arguments specified.

Steps to reproduce the behavior

Set up replication.yaml as stated with this:

S3_ ENDPOINT -> "https://s3-eu-west-1.amazonaws.com"
BUCKET -> "ngi-igenomes"
LOCAL_BUCKET -> any local bucket mc batch start ALIAS replication.yaml
mc --version

mc version RELEASE.2023-08-18T21-57-55Z (commit-id=acc25b8c5057e85dbca855f9d417f13de0603e60) Runtime: go1.19.12 linux/amd64 Copyright (c) 2015-2023 MinIO, Inc. License GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html

System information

Ubuntu 22.04 - amd64

vbob commented 2 months ago

Also facing this issue.

I want to sync from AWS open data https://registry.opendata.aws/ons-opendata-portal/

replication.yaml

replicate:
  apiVersion: v1
  source:
    type: s3 
    bucket: "ons-aws-prod-opendata"
    prefix: "" 
    endpoint: "https://ons-dl-prod-opendata.s3.amazonaws.com/"
    credentials:
      accessKey: "" 
      secretKey: "" 

  target:
    type: minio
    bucket: ons
    prefix: ""

mc batch start minio-local/ ./replication.yaml --debug --json

{
    "Code":"InvalidArgument",
    "Message":"Amazon S3 endpoint should be 's3.amazonaws.com'.",
    "Resource":"/minio/admin/v3/start-job",
    "RequestId":"17F14704E7EA3DB3",
    "HostId":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8"
}

Also tried with the s3.amazonaws.com and s3-sa-east-1.amazonaws.com with no luck:

"error": {
  "message": "Unable to start job",
  "cause": {
   "message": "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.",
   "error": {
    "Code": "PermanentRedirect",
    "Message": "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.",
    "BucketName": "",
    "Key": "",
    "RequestID": "17F147EA729BDE6F",
    "HostID": "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8",
    "Region": ""
   }
  },

vadmeste commented 2 months ago

@jhidalgo-lopez can you upgrade to the latest and try again ?

@vbob what is your MinIO version ?

dajeffers commented 2 weeks ago

I seem to be having this issue as well.

mc --version 
mcli version RELEASE.2024-10-29T15-34-59Z (commit-id=9f4659884dd45dca726ba38ee6bfacb2bf776eb8)
Runtime: go1.22.7 linux/amd64
Copyright (c) 2015-2024 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>

# mc mb play/testreplicate1
Bucket created successfully `play/testreplicate1`.
# mc mb play/testreplicate2
Bucket created successfully `play/testreplicate2`.
# cat /tmp/play-replicate.yaml 
replicate:
  apiVersion: v1
  # source of the objects to be replicated
  source:
    type: minio # valid values are "s3" or "minio"
    bucket: testreplicate1
    endpoint: "https://play.min.io"
    credentials:
      accessKey: ACCESSKEYREDACTED
      secretKey: SECRETKEYREDACTED

  target:
    type: minio # valid values are "s3" or "minio"
    bucket: testreplicate2
    endpoint: "local"
    prefix: "" # 'PREFIX' is optional

# mc batch start play /tmp/play-replicate.yaml
mcli: <ERROR> Unable to start job. Invalid arguments specified.

Seems to fail against both the MinIO Play deployment and the tenant-6.0.1 installations I run. Am I perhaps missing something with the syntax?

minio / mc