minio / mc

Unix like utilities for object store
https://min.io/download
GNU Affero General Public License v3.0
2.86k stars 548 forks source link

mc batch replication without S3 credentials #4862

Open jhidalgo-lopez opened 8 months ago

jhidalgo-lopez commented 8 months ago

Expected behavior

Considering BUCKET a public S3 bucket and the following replication.yaml:

replicate:
  apiVersion: v1
# SOURCE
  source:
    type: "s3"
    bucket: BUCKET
    prefix: ""
    endpoint: S3_ENDPOINT
    credentials:
      accessKey: ""
      secretKey: ""
# TARGET
  target:
    type: "minio"
    bucket: LOCAL_BUCKET

mc batch start ALIAS replication.yaml should work as mc mirror when setting up an mc alias such as mc alias set ALIAS S3_ENDPOINT without credentials: Replicating the public bucket using no S3 credentials.

Actual behavior

Task fails: mc: <ERROR> Unable to start job. Invalid arguments specified.

Steps to reproduce the behavior

Set up replication.yaml as stated with this:

vbob commented 2 months ago

Also facing this issue.

I want to sync from AWS open data https://registry.opendata.aws/ons-opendata-portal/

replication.yaml

replicate:
  apiVersion: v1
  source:
    type: s3 
    bucket: "ons-aws-prod-opendata"
    prefix: "" 
    endpoint: "https://ons-dl-prod-opendata.s3.amazonaws.com/"
    credentials:
      accessKey: "" 
      secretKey: "" 

  target:
    type: minio
    bucket: ons
    prefix: "" 

mc batch start minio-local/ ./replication.yaml --debug --json

{
    "Code":"InvalidArgument",
    "Message":"Amazon S3 endpoint should be 's3.amazonaws.com'.",
    "Resource":"/minio/admin/v3/start-job",
    "RequestId":"17F14704E7EA3DB3",
    "HostId":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8"
}

Also tried with the s3.amazonaws.com and s3-sa-east-1.amazonaws.com with no luck:

"error": {
  "message": "Unable to start job",
  "cause": {
   "message": "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.",
   "error": {
    "Code": "PermanentRedirect",
    "Message": "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.",
    "BucketName": "",
    "Key": "",
    "RequestID": "17F147EA729BDE6F",
    "HostID": "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8",
    "Region": ""
   }
  },
vadmeste commented 2 months ago

@jhidalgo-lopez can you upgrade to the latest and try again ?

@vbob what is your MinIO version ?