Cannot install package with files that have non-ascii filenames

[megies]

For me installation of a python package fails because some test files in it have non-ascii characters in filenames:

$ pip install https://github.com/megies/seishub.core/archive/quakeml.zip
Downloading/unpacking https://github.com/megies/seishub.core/archive/quakeml.zip
  Downloading quakeml.zip (unknown size): 3.5MB downloaded
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/commands/install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/req.py", line 1197, in prepare_files
    do_download,
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/req.py", line 1375, in unpack_url
    self.session,
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/download.py", line 582, in unpack_http_url
    unpack_file(temp_location, location, content_type, link)
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/util.py", line 621, in unpack_file
    unzip_file(filename, location, flatten=not filename.endswith(('.pybundle', '.whl')))
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/util.py", line 492, in unzip_file
    leading = has_leading_dir(zip.namelist()) and flatten
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/util.py", line 232, in has_leading_dir
    prefix, rest = split_leading_dir(path)
  File "/home/megies/python_seishub/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/util.py", line 216, in split_leading_dir
    path = str(path)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 85-87: ordinal not in range(128)

Storing debug log for failure in /home/megies/.pip/pip.log

The problematic files are some test files in there: https://github.com/megies/seishub.core/tree/quakeml/seishub/core/processor/tests/data/filesystem/special

I'm using pip 1.5.4 on Python 2.7.6.

Installation seems to run smoothly when commenting out this line: https://github.com/pypa/pip/blob/develop/pip/util.py#L228

[dholth]

We ran into a similar issue where the URLs (network) part of pip was operating in Unicode, while the filesystem part was operating in bytes. There may need to be a decode(sys.getdefaultfilesystemencoding()) [spelling?] in there somewhere.

[blueyed]

I am seeing this error when trying to install Django master via tox.

Django now ships a file with a unicode symbol: https://github.com/django/django/commit/bd059e3f8c6311dcaf8afe5e29ef373f7f84cf26, and the traceback from pip is:

Collecting https://github.com/django/django/archive/master.zip
  Downloading https://github.com/django/django/archive/master.zip (11.4MB)
  Exception:
  Traceback (most recent call last):
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/basecommand.py", line 232, in main
      status = self.run(options, args)
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/commands/install.py", line 339, in run
      requirement_set.prepare_files(finder)
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/req/req_set.py", line 355, in prepare_files
      do_download, session=self.session,
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/download.py", line 782, in unpack_url
      session,
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/download.py", line 671, in unpack_http_url
      unpack_file(from_path, location, content_type, link)
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/utils/__init__.py", line 648, in unpack_file
      flatten=not filename.endswith('.whl')
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/utils/__init__.py", line 528, in unzip_file
      leading = has_leading_dir(zip.namelist()) and flatten
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/utils/__init__.py", line 253, in has_leading_dir
      prefix, rest = split_leading_dir(path)
    File "…/pytest_django/.tox/python2.7-master-sqlite_file/lib/python2.7/site-packages/pip/utils/__init__.py", line 237, in split_leading_dir
      path = str(path)
  UnicodeEncodeError: 'ascii' codec can't encode character u'\u2297' in position 60: ordinal not in range(128)

(Initially reported for tox at https://bitbucket.org/hpk42/tox/issue/221/unicodeencodeerror-when-trying-to-install).

[blueyed]

Duplicate of #1140, and https://github.com/pypa/pip/pull/1473 has a fix.

[piotr-dobrogost]

@blueyed 1473 has a fix for .tar files and this bug is for .zip files. Maybe the fix for .zip is the same or very similar but these are two different bugs from user point of view.

[dstufft]

The underlying issue here appears to have been resolved, this works with Django. If this is still an issue for you please file a new bug with reproduction steps.

[ryansb]

I'm still having what I believe to be this same problem when installing ansible from the Github zip-archives. Reproduction steps:

OS: CentOS 7 Pip version: 9.0.1

[root@68768cc9b900 /]# pip --version
pip 9.0.1 from /usr/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg (python 2.7)
[root@68768cc9b900 /]# pip install https://github.com/ansible/ansible/archive/devel.zip 
Collecting https://github.com/ansible/ansible/archive/devel.zip
  Downloading https://github.com/ansible/ansible/archive/devel.zip (7.4MB)
    100% |################################| 7.4MB 215kB/s 
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "/usr/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "/usr/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 821, in unpack_url
    hashes=hashes
  File "/usr/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/download.py", line 663, in unpack_http_url
    unpack_file(from_path, location, content_type, link)
  File "/usr/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/__init__.py", line 599, in unpack_file
    flatten=not filename.endswith('.whl')
  File "/usr/lib/python2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/__init__.py", line 499, in unzip_file
    fp = open(fn, 'wb')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 87-90: ordinal not in range(128)

Even on the current master branch, it still seems like an issue:

[root@68768cc9b900 /]# python --version
Python 2.7.5
[root@68768cc9b900 /]# pip install https://github.com/pypa/pip/archive/master.zip 
Collecting https://github.com/pypa/pip/archive/master.zip
  Downloading https://github.com/pypa/pip/archive/master.zip (6.4MB)
    100% |################################| 6.4MB 237kB/s 
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
  Running setup.py install for pip ... done
Successfully installed pip-10.0.0.dev0
[root@68768cc9b900 /]# pip install https://github.com/ansible/ansible/archive/devel.zip
Collecting https://github.com/ansible/ansible/archive/devel.zip
  Downloading https://github.com/ansible/ansible/archive/devel.zip (7.4MB)
    100% |################################| 7.4MB 4.8MB/s 
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pip/basecommand.py", line 216, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/site-packages/pip/commands/install.py", line 254, in run
    requirement_set.prepare_files(finder)
  File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 367, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 609, in _prepare_file
    progress_bar=self.progress_bar)
  File "/usr/lib/python2.7/site-packages/pip/download.py", line 834, in unpack_url
    progress_bar=progress_bar
  File "/usr/lib/python2.7/site-packages/pip/download.py", line 674, in unpack_http_url
    unpack_file(from_path, location, content_type, link)
  File "/usr/lib/python2.7/site-packages/pip/utils/__init__.py", line 599, in unpack_file
    flatten=not filename.endswith('.whl')
  File "/usr/lib/python2.7/site-packages/pip/utils/__init__.py", line 499, in unzip_file
    fp = open(fn, 'wb')
UnicodeEncodeError: 'ascii' codec can't encode characters in position 87-90: ordinal not in range(128)
[root@68768cc9b900 /]# pip --version
pip 10.0.0.dev0 from /usr/lib/python2.7/site-packages (python 2.7)