aaugustin
commented
8 years ago For what it's worth, the following patch solves my problem:
diff -r -u -x '*.pyc' -w awscli.orig/customizations/s3/filegenerator.py awscli/customizations/s3/filegenerator.py
--- awscli.orig/customizations/s3/filegenerator.py 2015-11-15 08:50:45.000000000 +0100
+++ awscli/customizations/s3/filegenerator.py 2015-11-15 18:56:31.000000000 +0100
@@ -13,6 +13,7 @@
import os
import sys
import stat
+import unicodedata
from dateutil.parser import parse
from dateutil.tz import tzlocal
@@ -116,11 +117,12 @@
``FileInfo`` objects to send to a ``Comparator`` or ``S3Handler``.
"""
def __init__(self, client, operation_name, follow_symlinks=True,
- page_size=None, result_queue=None):
+ page_size=None, normalize_unicode=False, result_queue=None):
self._client = client
self.operation_name = operation_name
self.follow_symlinks = follow_symlinks
self.page_size = page_size
+ self.normalize_unicode = normalize_unicode
self.result_queue = result_queue
if not result_queue:
self.result_queue = queue.Queue()
@@ -167,6 +169,8 @@
"""
join, isdir, isfile = os.path.join, os.path.isdir, os.path.isfile
error, listdir = os.error, os.listdir
+ if self.normalize_unicode:
+ path = unicodedata.normalize('NFKC', path)
if not self.should_ignore_file(path):
if not dir_op:
size, last_update = get_file_stat(path)
@@ -185,6 +189,8 @@
listdir_names = listdir(path)
names = []
for name in listdir_names:
+ if self.normalize_unicode:
+ name = unicodedata.normalize('NFKC', name)
if not self.should_ignore_file_with_decoding_warnings(
path, name):
file_path = join(path, name)
diff -r -u -x '*.pyc' -w awscli.orig/customizations/s3/s3handler.py awscli/customizations/s3/s3handler.py
--- awscli.orig/customizations/s3/s3handler.py 2015-11-15 08:50:45.000000000 +0100
+++ awscli/customizations/s3/s3handler.py 2015-11-15 09:25:54.000000000 +0100
@@ -64,7 +64,8 @@
'grants': None, 'only_show_errors': False,
'is_stream': False, 'paths_type': None,
'expected_size': None, 'metadata_directive': None,
- 'ignore_glacier_warnings': False}
+ 'ignore_glacier_warnings': False,
+ 'normalize_unicode': False}
self.params['region'] = params['region']
for key in self.params.keys():
if key in params:
diff -r -u -x '*.pyc' -w awscli.orig/customizations/s3/subcommands.py awscli/customizations/s3/subcommands.py
--- awscli.orig/customizations/s3/subcommands.py 2015-11-15 08:50:45.000000000 +0100
+++ awscli/customizations/s3/subcommands.py 2015-11-15 18:18:23.000000000 +0100
@@ -301,12 +301,21 @@
}
+NORMALIZE_UNICODE = {
+ 'name': 'normalize-unicode', 'action': 'store_true',
+ 'help_text': (
+ 'Normalizes file names read from the local filesystem in unicode '
+ 'normal form KC. This is mainly useful when running on OS X.'
+ )
+}
+
+
TRANSFER_ARGS = [DRYRUN, QUIET, INCLUDE, EXCLUDE, ACL,
FOLLOW_SYMLINKS, NO_FOLLOW_SYMLINKS, NO_GUESS_MIME_TYPE,
SSE, STORAGE_CLASS, GRANTS, WEBSITE_REDIRECT, CONTENT_TYPE,
CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING,
CONTENT_LANGUAGE, EXPIRES, SOURCE_REGION, ONLY_SHOW_ERRORS,
- PAGE_SIZE, IGNORE_GLACIER_WARNINGS]
+ PAGE_SIZE, IGNORE_GLACIER_WARNINGS, NORMALIZE_UNICODE]
def get_client(session, region, endpoint_url, verify):
@@ -770,10 +779,12 @@
operation_name,
self.parameters['follow_symlinks'],
self.parameters['page_size'],
+ self.parameters['normalize_unicode'],
result_queue=result_queue)
rev_generator = FileGenerator(self._client, '',
self.parameters['follow_symlinks'],
self.parameters['page_size'],
+ self.parameters['normalize_unicode'],
result_queue=result_queue)
taskinfo = [TaskInfo(src=files['src']['path'],
src_type='s3',I'm not submitting it as a PR because it's missing at least tests and documentation. I'm mostly leaving it here in case others find it helpful.
Of course, feel free to use it as a starting point for fixing this issue if my approach doesn't seem too off base.
EDIT: just updated the patch to apply unicode normalization before sorting file names.
JordonPhillips
BenAbineriBubble
ishikawa
ASayre
salmanwaheed
Summary
aws s3 syncdoesn't play well with HFS+ unicode normalization on OS X. I suggest to add an option to normalize file names read locally in normal form C before doing anything with them.Reproduction steps
Create a file on S3 containing an accented character. For reasons that will become apparent later, do this on a Linux system.
Synchronize that file on a Mac.
Synchronize it back to S3.
At this point the file shows up twice in S3!
Why this happens
Unicode defines two normal forms — NFC and NFD — for some characters, typically accented characters which are common in Western European languages and even occur in English.
The documentation of unicodedata.normalize, the Python function that converts between the two forms, has a good explanation.
A quick illustration:
The default filesystem of OS X, HFS+, enforces something that resembles NFD. (Let's say I haven't encountered the difference yet.)
Pretty much everything else, including typing on a keyboard on Linux or OS X, uses NFC. I'm not sure about Windows.
Of course this is entirely HFS+'s fault, but since OS X is a popular system among your target audience, I hope you may have some interest in providing a solution to this problem.
What you can do about it
I think a
--normalize-unicodeoption (possibly with a better name) foraws s3 syncwould be useful. It would normalize file names read from the local filesystem withunicodedata.normalize('NFKC', filepath).Its primary purpose would be to interact with S3 on OS X and have file names in NFC form on S3, which is what the rest of the world expects and will cause the least amount of problems.
I don't know
aws cliwell enough to tell which other parts could use this option. I just encountered the problem when trying to replace "rsyncto file server" with "aws s3 syncto S3".FWIW
rsyncprovides a solution to this problem with the--iconvoption. A common idiom is--iconv=UTF8-MAC,UTF8when rsync'ing from OS X to Linux and--iconv=UTF8,UTF8-MACwhen rsync'ing from Linux to OS X.UTF8-MACis howrsynccalls the encoding of file names on HFS+.However this isn't a good API to tackle the specific problem I'm raising here. This API is about the encoding of file names. The bug is related to Unicode normalization. These are different concepts.
UTF8-MACmixes them.Thanks!