`aws sync` fails silently (but does take time)

[ghost]

(linux, bash) I have a command like the one below. It runs without error and without output but it does take a few seconds to run. It also does not upload or delete files that I know are outdated on the server. If I put in a switch error into the statement, then I do get errors. If I do aws s3 ls s3://myapp-contentdelivery-mobilehub-123456 I get the files listed. If I do aws s3 cp "./index.html" s3://myapp-contentdelivery-mobilehub-123456 the upload succeeds (so the permissions are OK).

I notice that putting quotes around the bucket in cp like aws s3 cp "./index.html" "s3://myapp-contentdelivery-mobilehub-123456" then it yields Error: Invalid argument type which is silly too, but I tried it both ways in sync and there's no difference in the operation.

aws s3 sync . "s3://myapp-contentdelivery-mobilehub-123456" --dryrun --delete --exclude "*" --include
 "./preview.html" --include "./index.html" --include "./favicon.ico"  --include "./fonts/fontawesome-
webfont.eot" --include "./fonts/FontAwesome.otf" --include "./fonts/fontawesome-webfont.ttf" --include 
"./fonts/fontawesome-webfont.woff2" --include "./fonts/fontawesome-webfont.woff" --include 
"./fonts/fontawesome-webfont.svg" --include "./css/font-awesome.css" --include "./css/font-
awesome.min.css"

I tried a lot of variations and a few times I figured it out by getting rid of the leading ./ or that there were too many files listed, or that it doesn't like dashes in the name, or that it doesn't like the quotes around the include file names (but that would be silly - what about file names with spaces? and anyway --exclude uses them, so that would be inconsistent).

[JordonPhillips]

Could you append --debug to the command and post the response here?

[ghost]

That helps a little! I'm happy to sent you the whole file privately to an email address or something but this looks relevant to the problem:

...
2016-01-11 13:24:03,641 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.9.17 Python/2.7.6 Linux/3.19.0-43-generic botocore/1.3.17
2016-01-11 13:24:03,641 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI:  ['s3', 'sync', '.', 's3://myapp-contentdelivery-mobilehub-123456', '--dryrun', '--delete', '--exclude', '*', '--include', '"./preview.html"', '--include', '"./index.html"', '--include', '"./favicon.ico"', ...
...
2016-01-11 13:24:08,953 - MainThread - awscli.customizations.s3.filters - DEBUG - myapp-contentdelivery-mobilehub-123456/preview.html did not match clude filter: /home/me/Projects/myApp/myAppMobile/*
2016-01-11 13:24:08,953 - MainThread - awscli.customizations.s3.filters - DEBUG - myapp-contentdelivery-mobilehub-123456/preview.html matched exclude filter: myapp-contentdelivery-mobilehub-123456/*
2016-01-11 13:24:08,954 - MainThread - awscli.customizations.s3.filters - DEBUG - myapp-contentdelivery-mobilehub-123456/preview.html did not match clude filter: /home/me/Projects/myApp/myAppMobile/"./preview.html"
2016-01-11 13:24:08,954 - MainThread - awscli.customizations.s3.filters - DEBUG - myapp-contentdelivery-mobilehub-123456/preview.html did not match clude filter: myapp-contentdelivery-mobilehub-123456/"./preview.html"
2016-01-11 13:24:08,954 - MainThread - awscli.customizations.s3.filters - DEBUG - myapp-contentdelivery-mobilehub-123456/preview.html did not match clude filter: /home/me/Projects/myApp/myAppMobile/"./index.html"
2016-01-11 13:24:08,955 - MainThread - awscli.customizations.s3.filters - DEBUG - myapp-contentdelivery-mobilehub-123456/preview.html did not match clude filter: myapp-contentdelivery-mobilehub-123456/"./index.html"
2016-01-11 13:24:08,994 - MainThread - awscli.customizations.s3.filters - DEBUG - =myapp-contentdelivery-mobilehub-123456/preview.html final filtered status, should_include: False
2016-01-11 13:24:08,994 - MainThread - awscli.customizations.s3.executor - DEBUG - Queueing end sentinel for worker thread (priority: 11)
...

Suggest it doesn't like the quotes around include paths, but how the file paths are read in, but how should I properly quote paths then?

[JordonPhillips]

@MrAndMrsK Try leaving out the ./. What you're providing is a very limited regular expression, which operates on the object key. ./ is a valid key name/part, and the CLI is not seeing that.

[ghost]

Taking out just the ./ still has a problem. /home/me/Projects/myApp/myAppMobile/"preview.html"

I can take out the quotes, but what about paths that need escaping?

[JordonPhillips]

It's strange that the quotes are being passed in literally, that shouldn't be happening. What distro are you running, and what shell are you using? I can't reproduce on OSX using zsh.

$ aws --version
aws-cli/1.9.18 Python/2.7.10 Darwin/14.5.0 botocore/1.3.18
$ ls
CHANGELOG.rst           MANIFEST.in             awscli.egg-info         requirements.txt        setup.cfg               tests
CONTRIBUTING.rst        README.rst              bin                     requirements26.txt      setup.py                tmp.log
LICENSE.txt             awscli                  doc                     scripts                 tox.ini
$ aws s3 ls s3://test-bucket
$ aws s3 sync . s3://phjordon-test-bucket/ --exclude "*" --include "requirements.txt" --debug
...
Arguments entered to CLI: ['s3', 'sync', '.', 's3://phjordon-test-bucket/', '--exclude', '*', '--include', 'requirements.txt', '--debug']
...
upload: ./requirements.txt to s3://phjordon-test-bucket/requirements.txt

[ghost]

(linux, bash) CLI version: aws-cli/1.9.17 Python/2.7.6 Linux/3.19.0-43-generic botocore/1.3.17

This is Ubuntu-based Elementary.

(Also, I meant to point out that link about "very limited regular expression" doesn't mention anything about regular expressions or the . wildcard. Perhaps that's a doc bug?)

[JordonPhillips]

@MrAndMrsK Sorry for the confusion, I meant regular expression in the general sense, not that it complies with PCRE or some such. I'll try reproducing on a linux instance, but in the meantime leaving out the escaping quotes should get you running.

[JordonPhillips]

@MrAndMrsK Sorry for the long delay. I've tested against a few linux distros and I can't seem to reproduce. Are you still seeing the issue?

[ghost]

Yes, I believe it still happens but I just got rid of the quotes and just have to be careful about using filenames that don't need escaping, I guess.

[JordonPhillips]

Hmm.... Do you perhaps have any odd bash configurations (perhaps elementary ships some)? To get the command line parameters we're calling sys.argv which depends entirely on the shell in use. Could you see if the problem persists in another shell? (cat /etc/shells for a list of installed shells.) Though I couldn't get it to reproduce in Ubuntu, I'll spin up an Elementary VM (sadly no ami :cry:) to see if that's a more fruitful path.

[JordonPhillips]

Fresh elementary vm also does not reproduce the input error. Something is causing your shell to pass in the quotes as if they were escaped, either before it gets to python, or as python is passing that data to us. Given that quotes are perfectly valid characters in both filenames and S3 keys, I don't think there's anything we can do on our end.

I'm going to go ahead and close this issue since I don't think the root cause is on our end. That said, please update if you find out any more information that might help diagnose the issue or if you find the solution.