Suggestion: We have the "mirror" sub command, but what about deleting left over files on the Remote ?

[ghost]

at the moment mirror is pretty good. mc mirror --force --json $LOCAL $REMOTE

But in the above script, if i deleted a file locally, up on s3 (or mini server) it will not also delete the file. It leaves it there on s3. If would be great if a second flag can be set to force deletes. The use cases are many for this. For me, i am in charge of a code base, and its got many binaries that others need. At the moment there is no way (as fas as i can see) to do a miror UP to s3, which also deletes left overs.

Looking forward to your responses on this ...

thanks Joe

[vadmeste]

@gedw99, --remove flag is recently added to mirror, it removes remote files when they are not found in local.

[joeblew99]

@vadmeste Thanks for quick response. Awesome it works, but i found a bug.

Bug: It will never delete a Folder on S3 that is empty, but reports that it did.

Reproduction: On S3 (Remote) i have a folder called "aa", and inside that a file called "bb.pdf". On Local, "aa" and "bb" do not exist.

Expectations: What it does is deletes "bb.pdf", but never deletes "aa", no matter how many runs i give it. For example, when i run the same script 10 times it always tells me it deleted "aa", yet it never does. What it should do is delete the folder "aa" & file "bb.pdf" on S3, because neither are on local.

This might sound small, but it confuses the tooling i have built on top. Its telling fibs :) Could you have a look at this and let me know if you agree it merits fixing please ? BTW, I have not tried the reverse as a test. to mirror from S3 --> local to see if it makes the same logical error. Worth testing also !!!

My bash script (running MC off the head from github)

#!/bin/bash

# minio
go get -u github.com/minio/mc

# Get binary tools

LOCAL=$GOPATH"/src/bitbucket.org/XXX/YYY/"
REMOTE=s3/YYY/

echo "LOCAL: " $LOCAL
echo "REMOTE: " $REMOTE

echo "# ls LOCAL"
ls -l $LOCAL

echo "# ls REMOTE"
mc ls $REMOTE

echo "# mirror up to REMOTE"
mc mirror --force --remove $LOCAL $REMOTE 

See this output:

x-MacBook-Pro:backend apple$ ./tool-deps-set-YYY.sh 
LOCAL:  /Users/apple/workspace/go/src/bitbucket.org/XXX/YYY/
REMOTE:  s3/YYY/
# ls LOCAL
total 7824
drwxr-xr-x   4 apple  staff      136 Mar 24 18:24 chrome-portable
drwxr-xr-x   5 apple  staff      170 Mar 23 17:56 cloudBerry-explorer
drwxr-xr-x   4 apple  staff      136 Mar  2 16:05 filezilla-portable
drwxr-xr-x   6 apple  staff      204 Mar 24 11:52 securitystuff
-rw-r--r--@  1 apple  staff  4003825 Mar 24 12:34 tinker-tool.dmg
drwxr-xr-x  15 apple  staff      510 Mar 23 18:05 windows-nuc-machine-aboutyou
# ls REMOTE
[2016-04-21 17:29:38 CEST] 3.8MiB tinker-tool.dmg
[2016-04-22 11:56:59 CEST]     0B aa/
[2016-04-22 11:56:59 CEST]     0B chrome-portable/
[2016-04-22 11:56:59 CEST]     0B cloudBerry-explorer/
[2016-04-22 11:56:59 CEST]     0B filezilla-portable/
[2016-04-22 11:56:59 CEST]     0B securitystuff/
[2016-04-22 11:56:59 CEST]     0B windows-nuc-machine-aboutyou/
# mirror up to REMOTE
Removed ‘s3/YYY/aa’.
Removed ‘s3/YYY/aa/bb.pdf’.
x-MacBook-Pro:backend apple$ ./tool-deps-set-YYY.sh 
LOCAL:  /Users/apple/workspace/go/src/bitbucket.org/XXX/YYY/
REMOTE:  s3/YYY/
# ls LOCAL
total 7824
drwxr-xr-x   4 apple  staff      136 Mar 24 18:24 chrome-portable
drwxr-xr-x   5 apple  staff      170 Mar 23 17:56 cloudBerry-explorer
drwxr-xr-x   4 apple  staff      136 Mar  2 16:05 filezilla-portable
drwxr-xr-x   6 apple  staff      204 Mar 24 11:52 securitystuff
-rw-r--r--@  1 apple  staff  4003825 Mar 24 12:34 tinker-tool.dmg
drwxr-xr-x  15 apple  staff      510 Mar 23 18:05 windows-nuc-machine-aboutyou
# ls REMOTE
[2016-04-21 17:29:38 CEST] 3.8MiB tinker-tool.dmg
[2016-04-22 11:57:12 CEST]     0B aa/
[2016-04-22 11:57:12 CEST]     0B chrome-portable/
[2016-04-22 11:57:12 CEST]     0B cloudBerry-explorer/
[2016-04-22 11:57:12 CEST]     0B filezilla-portable/
[2016-04-22 11:57:12 CEST]     0B securitystuff/
[2016-04-22 11:57:12 CEST]     0B windows-nuc-machine-aboutyou/
# mirror up to REMOTE
Removed ‘s3/YYY/aa’.

[krishnasrinivas]

@joeblew99 did you create aa/ using s3-UI-browser? answer to this question will give more clues. The output of mc ls s3/YYY/aa/ will also give more clues ... note the / at the end

[joeblew99]

@krishnasrinivas Ok i just tried it again with FIRST creating "aa" folder & "bb" file on S3.

I got the same bug as the first time..

Here is my test bash script to help:

#!/bin/bash

# minio
go get -u github.com/minio/mc

# Get binary tools

LOCAL=$GOPATH"/src/bitbucket.org/XXX/YYY/"
REMOTE=s3/YYY/

echo "LOCAL: " $LOCAL
echo "REMOTE: " $REMOTE

#echo "# ls LOCAL"
#ls -l $LOCAL

#echo "# ls REMOTE"
#mc ls $REMOTE

echo "Doing DIff of Local against Remote ..."
mc diff $LOCAL $REMOTE

read -n1 -r -p "Press 'y' to start mirror with Remote (S3) ? " key

if [ "$key" = 'y' ]; then
    echo ""
    echo "# Mirroring up to REMOTE, and deleting any files on Remote(S3) that are not on local."
    mc mirror --force --remove $LOCAL $REMOTE 
else
    echo ""
    echo "No action taken."
fi

[joeblew99]

I also tried LOCALLY creating folder "aa" and file "bb" FIRST. then i ran the same bash script. "aa" and "bb" now exist on S3 then i deleted "b" locally. Does it leave "aa" on S3 incorrectly ? Was curious if the bug still exists in this scenario. Using the same bash script as above.

Terminal:

x-MacBook-Pro:backend apple$ ./script.sh 
LOCAL:  /Users/apple/workspace/go/src/bitbucket.org/XXX/YYY/
REMOTE:  s3/YYY/
Doing DIff of Local against Remote ...
‘https://s3.amazonaws.com/YYY/aa’ - only in second.
‘https://s3.amazonaws.com/YYY/aa/b.exe’ - only in second.
Press 'y' to start mirror with Remote (S3) ? y
# Mirroring up to REMOTE, and deleting any files on Remote(S3) that are not on local.
Removed ‘s3/YYY/aa’.
Removed ‘s3/YYY/aa/b.exe’.

The result is that on S3 "a" remains. "b" is gone. But MC is telling me that "a" is gone - Its telling fibs :)

So its doing the exact same thing, irrespective of where i create the "a" and "b" file.

[krishnasrinivas]

@joeblew99 you mention "a" "b" and "aa" "bb" it's confusing please clarify. Also I will still need answers to my previous questions:

@joeblew99 did you create aa/ using s3-UI-browser? answer to this question will give more clues. The output of mc ls s3/YYY/aa/ will also give more clues ... note the / at the end

because I suspect "aa" was created using aws S3 browser and that is what is causing problems. If you clarify we can explore more on what is happening.

[krishnasrinivas]

update: aa/ was created using AWS S3-browser. Need to discuss if mc should support such a scenario.

[harshavardhana]

Closing as duplicate of #1716