zipfile.extractall fails in Posix shell with utf-8 filename

[be1526b9-3573-4f3d-abe3-d392fe59c2a9]

BPO	20329
Nosy	@ncoghlan, @vstinner, @bitdancer, @serhiy-storchaka, @csabella
Files	test_ut8.zip: Zip where filenames are in UTF-8

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['extension-modules', 'type-bug'] title = 'zipfile.extractall fails in Posix shell with utf-8 filename' updated_at = user = 'https://bugs.python.org/LaurentMazuel' ``` bugs.python.org fields: ```python activity = actor = 'vstinner' assignee = 'none' closed = False closed_date = None closer = None components = ['Extension Modules'] creation = creator = 'Laurent.Mazuel' dependencies = [] files = ['33589'] hgrepos = [] issue_num = 20329 keywords = [] message_count = 10.0 messages = ['208648', '208655', '208755', '208817', '208858', '208859', '212349', '308345', '308350', '308568'] nosy_count = 6.0 nosy_names = ['ncoghlan', 'vstinner', 'r.david.murray', 'Laurent.Mazuel', 'serhiy.storchaka', 'cheryl.sabella'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue20329' versions = ['Python 3.3'] ```

[be1526b9-3573-4f3d-abe3-d392fe59c2a9]

Hello,

Considering a zip file which contains utf-8 filenames (as uploaded zip file), the following code fails if launched in a Posix shell.

>>> with zipfile.ZipFile("test_ut8.zip") as fd:
...     fd.extractall()
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/opt/python/3.3/lib/python3.3/zipfile.py", line 1225, in extractall
    self.extract(zipinfo, path, pwd)
  File "/opt/python/3.3/lib/python3.3/zipfile.py", line 1213, in extract
    return self._extract_member(member, path, pwd)
  File "/opt/python/3.3/lib/python3.3/zipfile.py", line 1276, in _extract_member
    open(targetpath, "wb") as target:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 10-14: ordinal not in range(128)

With shell: $ locale LANG=POSIX ...

But filesystem is not encoding dependant. On a Unix system, filename are only bytes, there is no reason to refuse to unzip a zip file (in fact, "unzip" command line don't fail to unzip the file in a Posix shell).

Since "open" can take "bytes" filename, changing the line 1276 from

open(targetpath) to: open(targetpath.encode("utf-8"))

fixes the problem.

zipfile should not care about the encoding of the filename and should use the bytes sequence filename extracted directly from the bytes sequence of the zipfile. Having "ZipInfo.filename" as a string (and not bytes) is great for an API, but is not needed to open/write a file on the disk. Then, ZipInfo should store the direct bytes sequences of filename as a "bytes_filename" field and use it in the "open" of "extract".

In addition, considering the patch of bug 10614, the right patch could use the new "ZipInfo.encoding" field:

open(targetpath.encode(member.encoding))

[bitdancer]

If you live in a current-posix world, this might make sense. However, one can also argue that the filename should be *transcoded* from the tarfile encoding to the local FS filename encoding, which I believe is what we are currently doing. Which, if you are using POSIX as the locale, will fail a lot. If you use a sensible modern locale that includes utf-8, you wouldn't have a problem.

Unfortunately, the reality is probably that sometimes you want one behavior and sometimes you want the other :(

Encoding using member.encoding is probably wrong, though. If you are trying to preserve the original bytes, is is probably best do so, and not assume that the tarfile encoding field is valid.

I'm adding Victor Stinner to nosy: he's thought about these issues much more deeply than I have. The answer may be that we will only support transcoding filenames in our tarfile module...and certainly it looks like doing anything else, even if we want to, would be a new feature.

[be1526b9-3573-4f3d-abe3-d392fe59c2a9]

Thanks for your answer.

I think you can't transcode internal zip filenames to FS encoding. Actually, in Unix the FS only stores bytes for filename, there is no "FS encoding". Then, if you change your locale, the filename printed will change too in your console. If you transcode filename using the current locale, unzipping twice the same file with two different locales will lead to two different files, which is not (I think) you are intending for. The problem will not arise in Windows (NTFS is UTF-16) nor MAC OSX (UTF-8)

Moreover, a simple "unzip" works like a charm. It doesn't care about encoding or current locale and extract the file using the initial bytes in the zip. Unzipping twice with the two different locales creates only one file.

An interesting link (even if it is not an official reference): http://unix.stackexchange.com/questions/2089/what-charset-encoding-is-used-for-filenames-and-paths-on-linux

[bitdancer]

Believe me, we are *well* aware of the issue that linux stores filenames as bytes.

I agree that the inability to always transcode is an issue. That's why I'd like the opinion of someone who has studied this problem in more depth.

[ncoghlan]

The POSIX locale tells Python 3 to use ASCII for all operating system interfaces, including the standard streams. This is an antiquated behaviour in the POSIX spec that Python 3 doesn't currently work around.

bpo-19977 is a proposal to work around this limitation by default.

As an immediate workaround, it's possible to either set PYTHONIOENCODING explicitly so Python ignores the incorrect encoding claims from the OS, or else to do your own encoding and write directly to the sys.stdout.buffer binary interface.

Python 3.4 also allows setting *just* the default error handler for the streams, while still getting the encoding from the OS.

[ncoghlan]

My apologies, I completely misread the issue and thought it was related to displaying file names, rather than opening them.

I believe Python 3.4 includes some changes in this area - are you in a position to retry this on the latest 3.4 beta release?

[be1526b9-3573-4f3d-abe3-d392fe59c2a9]

Thank for your answer.

Unfortunately, I cannot test easily python 3.4 for now. But I have downloaded the source code and "diff" from 3.3 to 3.4 the "zipfile" module and see no difference relating to this problem. I can be wrong, maybe if some core improvement of Python may change something?

[csabella]

I created an environment under 3.3.1 in which this error was still occurring, but within that same environment, it is not occurring for 3.7. I believe this can be closed.

[vstinner]

I created an environment under 3.3.1 in which this error was still occurring, but within that same environment, it is not occurring for 3.7. I believe this can be closed.

Python 3.7 now uses the UTF-8 encoding when the LC_CTYPE locale is POSIX (PEP-538, PEP-540). You should still be able to reproduce the bug with a locale with an encoding different than UTF-8.

Moreover, I understand that Python 3.6 is still affected by the bug.

I don't think that we can fix this bug, sadly. But I'm happy to see that the PEP-538 and PEP-540 are already useful!

[vstinner]

I don't think that we can fix this bug, sadly. But I'm happy to see that the PEP-538 and PEP-540 are already useful!

Oops, I mean "we cannot *close* this bug" (right now). Sorry.

I mean that IMHO we still have to fix the bug.