mc cp error

[krishnasrinivas]

@vu3jej reported this:

So I was just trying out Minio Client on AWS EMR to copy bunch of files over to S3 like this
mc cp *.gz s3://jitesh-bucket/
and it was completed successfully!
I noticed that there was a slight discrepancy between the size in the EMR instance and the size in S3 for one of the files
EMR
-rw-rw-r-- 1 hadoop hadoop 4537765888 Dec 10 06:36 leads.lps_leads.gz
S3
2015-12-10 07:39:14 4522037248 leads.lps_leads.gz
I copied the same file using aws cli client after that and the sizes were the same!

mc version:

Version: 2015-12-06T01:59:23Z Release-tag: OFFICIAL.2015-12-06T01-59-23Z Commit-id: 9fed1d73eed07adf4406b5bc7cbdac58243a4629

[krishnasrinivas]

krishna@mcbug:~$ mc rm https://peppadog.s3.amazonaws.com/5gfile
mc: <ERROR> Unable to remove ‘https://peppadog.s3.amazonaws.com/5gfile’. Access Denied
krishna@mcbug:~$ mc rm https://peppadog.s3.amazonaws.com/5gfile
mc: <ERROR> Unable to stat. Access Denied
krishna@mcbug:~$ mc rm https://peppadog.s3.amazonaws.com/5gfile
Removed ‘https://peppadog.s3.amazonaws.com/5gfile’.
krishna@mcbug:~$ 

this might be related. maybe what happens is sometimes it takes https://s3.amazonaws.com creds and sometimes it takes https://peppadog.s3.amazonaws.com ?

[harshavardhana]

This was perhaps an important bug which surfaced . It was an oversight on my part and an important race was missed. Thanks to @krishnasrinivas for pointing it out on our new minio-go changes.

The race would happen so that all parts indeed have uploaded fine but the complete multipart 'slice' was never updated - this led to an issue where the complete multipart was issued on a multipart operation with additional parts missing.

For example if you have uploaded - 1, 2, 3, 4, 5 ,6, 7, 8, 9, 10 parts but you only have 1, 2, 3, 4, 5, 6, 7, 8 in your local complete multipart list. If you end up issuing a complete multipart upload on this list. Amazon S3 would create a file only comprising on 8parts not 10parts. Since final concatenation of parts is based on the complete multipart list.

New changes in minio-go will have this issue fixed - https://github.com/minio/minio-go/pull/250

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.