Open jhidalgo-lopez opened 8 months ago
Also facing this issue.
I want to sync from AWS open data https://registry.opendata.aws/ons-opendata-portal/
replication.yaml
replicate:
apiVersion: v1
source:
type: s3
bucket: "ons-aws-prod-opendata"
prefix: ""
endpoint: "https://ons-dl-prod-opendata.s3.amazonaws.com/"
credentials:
accessKey: ""
secretKey: ""
target:
type: minio
bucket: ons
prefix: ""
mc batch start minio-local/ ./replication.yaml --debug --json
{
"Code":"InvalidArgument",
"Message":"Amazon S3 endpoint should be 's3.amazonaws.com'.",
"Resource":"/minio/admin/v3/start-job",
"RequestId":"17F14704E7EA3DB3",
"HostId":"dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8"
}
Also tried with the s3.amazonaws.com
and s3-sa-east-1.amazonaws.com
with no luck:
"error": {
"message": "Unable to start job",
"cause": {
"message": "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.",
"error": {
"Code": "PermanentRedirect",
"Message": "The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.",
"BucketName": "",
"Key": "",
"RequestID": "17F147EA729BDE6F",
"HostID": "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8",
"Region": ""
}
},
@jhidalgo-lopez can you upgrade to the latest and try again ?
@vbob what is your MinIO version ?
Expected behavior
Considering BUCKET a public S3 bucket and the following replication.yaml:
mc batch start ALIAS replication.yaml
should work asmc mirror
when setting up an mc alias such asmc alias set ALIAS S3_ENDPOINT
without credentials: Replicating the public bucket using no S3 credentials.Actual behavior
Task fails:
mc: <ERROR> Unable to start job. Invalid arguments specified.
Steps to reproduce the behavior
Set up replication.yaml as stated with this:
mc batch start ALIAS replication.yaml
mc --version
mc version RELEASE.2023-08-18T21-57-55Z (commit-id=acc25b8c5057e85dbca855f9d417f13de0603e60) Runtime: go1.19.12 linux/amd64 Copyright (c) 2015-2023 MinIO, Inc. License GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html
System information
Ubuntu 22.04 - amd64