mcli: Unable to prepare URL for copying. Unable to guess the type of copy operation.

paramazo commented 6 months ago

Expected behavior

The folder in a bucket should be moved into another bucket

Actual behavior

The folder was not moved, mc results with error:

mc mv --recursive \<bucket1>/\<folder> \<bucket2>/\<folder> mcli: \<ERROR> Unable to prepare URL for copying. Unable to guess the type of copy operation.

Other folders in the same folder structure was moved this way successful.

In an earlier version of mc, the error was

mcli: \<ERROR> Unable to validate source

Steps to reproduce the behavior

Unknown why some folders cant be moved

mc --version

mcli version RELEASE.2024-03-30T15-29-52Z (commit-id=9f8147bf0e037730077a1b3baef25e53181099b0) Runtime: go1.21.8 linux/amd64

minio version RELEASE.2023-03-20T20-16-18Z (commit-id=05444a0f6af8389b9bb85280fc31337c556d4300) Runtime: go1.19.7 linux/amd64

System information

Debian 11.5 Bullseye

paramazo commented 6 months ago

Tested mc mirror but it does not copy anything from the affected folder and does not result with Error and ends after 2 minutes. The folder i want to copy is 101GB big with 1.8 million files.

Other smaller folders in the same bucket works with mc mv and with mc mirror.

paramazo commented 6 months ago

Tested to copy the folder from minio server to my local system with mc cp -r, same error. This folder can not be copied, mirrored or moved in any way. Other smaller folders work, so obviously it is a bug in MinIO with very big data. This is a gamebreaker and very critical and if this behaviour is not supported, we need to change our environment and switch to another solution.

paramazo commented 6 months ago

The data in our minio are now in a minio folder structure and as we can not access the contents via mcli/awscli/winscp, we have a huge data loss? Can anyone help here? Where do we find support?

ds-colligence commented 5 months ago

From what I can understand, it seems like when the bucket has hyphens in it, it is failing. At least that seems to be the case for me.

mmedic commented 5 months ago

I can confirm the same issue with a bucket with hypens in it

jiuker commented 5 months ago

@mmedic Could you share a script to let us to reproduce that?

hypesystem commented 5 months ago

I was wracking my brain because I got this error, too. It turns out it appears when the source file does not exist

mratibb commented 4 months ago

I had the same issue, and apparently it was a certificate issue. Either add 'insecure' option to your command, or add the certificate to the host/container you running the command on.

Xun66 commented 4 months ago

I had the same issue with two mc commands.

Command 1:

mc alias set myminio $MINIO_ENDPOINT $MINIO_ACCESS_KEY $MINIO_SECRET_KEY
mc cp --recursive myminio/<folder> <local>/<folder>
# mc: <ERROR> Unable to prepare URL for copying. Unable to guess the type of copy operation.

Looks like the MINIO_ACCESS_KEY and MINIO_SECRET_KEY were empty. Once I filled in the correct values, everything worked smoothly.

Command 2:

mc cp --recursive <local>/<folder> myminio/<folder>
# mc: <ERROR> Unable to prepare URL for copying. Unable to guess the type of copy operation.

Turns out, the issue was that <local>/<folder> didn't exist. In earlier versions of mc, this would've triggered an error message saying: mc: <ERROR> Unable to validate source <local>/<folder>: Object does not exist.

mc version

It looks like this change was introduced in PR #4710. From version mc version RELEASE.2023-10-14T01-57-03Z (commit-id=d158b9a478a6a5a74795f01097d069be82edfff6).

my test result:

$ docker run -it --rm minio/mc:RELEASE.2023-10-14T01-57-03Z cp -r /foo/ myio/bar 
mc: Configuration written to `/root/.mc/config.json`. Please update your access credentials.
mc: Successfully created `/root/.mc/share`.
mc: Initialized share uploads `/root/.mc/share/uploads.json` file.
mc: Initialized share downloads `/root/.mc/share/downloads.json` file.
mc: <ERROR> Unable to prepare URL for copying. Unable to guess the type of copy operation.

$ docker run -it --rm minio/mc:RELEASE.2023-10-04T06-52-56Z cp -r /foo/ myio/bar   
mc: Configuration written to `/root/.mc/config.json`. Please update your access credentials.
mc: Successfully created `/root/.mc/share`.
mc: Initialized share uploads `/root/.mc/share/uploads.json` file.
mc: Initialized share downloads `/root/.mc/share/downloads.json` file.
mc: <ERROR> Unable to validate source `/foo/`: Object does not exist

obarisk commented 3 months ago

I have the same issue. With a very small file (28 Bytes).

I can replicate the issue by

use golang PutObject to create a small file like xxx-1/7f9d85cf-7f48-408c-9649-f4c1ba472886/small.json
use mc cli (windows) to copy the object to local right after the object created.

mc: Unable to prepare URL for copying. Unable to guess the type of copy operation.
mc ls can find the object (small.json)
mc cat the object with redirect to local/small.json (works)
mc cp works after a couple hours.

cli version

mc version RELEASE.2024-05-03T11-21-07Z (commit-id=b471de8d1882cae21ca84e98d56c2a5e2c321164)
Runtime: go1.22.2 windows/amd64
Copyright (c) 2015-2024 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>

server version

minio version RELEASE.2024-06-13T22-53-53Z (commit-id=20960b6a2ddb9594ee418035b3c7c7fe92ae6a12)
Runtime: go1.22.4 linux/amd64
License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
Copyright: 2015-2024 MinIO, Inc.

XoseRamon commented 1 month ago

I had this very same issue when the source folder (uploading files to a minIO instance) contains hyphens. Removing the hyphens in the source folder fixed the issue. The error message is not very helpful.

jiuker commented 1 month ago

@XoseRamon Like what name?

XoseRamon commented 1 month ago

@jiuker I had something like mc cp test-data/*.gzip minio/test-bucket which threw an error and changing it to mc cp testdata/*.gzip minio/test-bucket worked. I am using:

mc version RELEASE.2024-08-17T11-33-50Z (commit-id=2c94b8d03979ed96a6379ecabc3435b271398efc)
Runtime: go1.22.6 linux/amd64
Copyright (c) 2015-2024 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>

jiuker commented 1 month ago

@jiuker I had something like mc cp test-data/*.gzip minio/test-bucket which threw an error and changing it to mc cp testdata/*.gzip minio/test-bucket worked. I am using:
mc version RELEASE.2024-08-17T11-33-50Z (commit-id=2c94b8d03979ed96a6379ecabc3435b271398efc)
Runtime: go1.22.6 linux/amd64
Copyright (c) 2015-2024 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>

What's the cp? remote to local?

XoseRamon commented 1 month ago

@jiuker I had something like mc cp test-data/*.gzip minio/test-bucket which threw an error and changing it to mc cp testdata/*.gzip minio/test-bucket worked. I am using:
mc version RELEASE.2024-08-17T11-33-50Z (commit-id=2c94b8d03979ed96a6379ecabc3435b271398efc)
Runtime: go1.22.6 linux/amd64
Copyright (c) 2015-2024 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
What's the cp? remote to local?

local to remote

jiuker commented 1 month ago

@jiuker I had something like mc cp test-data/*.gzip minio/test-bucket which threw an error and changing it to mc cp testdata/*.gzip minio/test-bucket worked. I am using:
mc version RELEASE.2024-08-17T11-33-50Z (commit-id=2c94b8d03979ed96a6379ecabc3435b271398efc)
Runtime: go1.22.6 linux/amd64
Copyright (c) 2015-2024 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
What's the cp? remote to local?
local to remote

Ok, I can't replicate this. Please share the quick bashScript to generate the dir test-data. And then I will follow this.

keplersj commented 2 days ago

I'm seeing this issue locally and in our Continuous Integration environment when using the mc cp --recursive command to copy local files to a bucket whose name contains a hyphen.

We did not see this failure before the October 2nd release of mc: https://github.com/minio/mc/releases/tag/RELEASE.2024-10-02T08-27-28Z.

minio / mc