search

pypa / pip

The Python package installer
https://pip.pypa.io/
MIT License
9.45k stars 3k forks source link

pip install codepage error (sqlalchemy,windows) #4110

Closed Goblenus closed 7 years ago

Goblenus commented 7 years ago

I have changed code page in my cmd to 65001 (it's utf-8), and while installing sqlalchemy by pip:

c:\Program Files\Python35\Scripts>pip3.exe install sqlalchemy

I obtained next error:

Collecting sqlalchemy
  Using cached SQLAlchemy-1.1.3.tar.gz
Installing collected packages: sqlalchemy
  Running setup.py install for sqlalchemy ... error
Exception:
Traceback (most recent call last):
  File "c:\program files\python35\lib\site-packages\pip\compat\__init__.py", line 73, in console_to_str
    return s.decode(sys.__stdout__.encoding)
  File "c:\program files\python35\lib\encodings\cp65001.py", line 17, in decode
    return codecs.code_page_decode(65001, input, errors, True)
UnicodeDecodeError: 'CP_UTF8' codec can't decode bytes in position 0--1: No mapping for the Unicode character exists in the target code page.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\program files\python35\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "c:\program files\python35\lib\site-packages\pip\commands\install.py", line 342, in run
    prefix=options.prefix_path,
  File "c:\program files\python35\lib\site-packages\pip\req\req_set.py", line 784, in install
    **kwargs
  File "c:\program files\python35\lib\site-packages\pip\req\req_install.py", line 878, in install
    spinner=spinner,
  File "c:\program files\python35\lib\site-packages\pip\utils\__init__.py", line 676, in call_subprocess
    line = console_to_str(proc.stdout.readline())
  File "c:\program files\python35\lib\site-packages\pip\compat\__init__.py", line 75, in console_to_str
    return s.decode('utf_8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 34: invalid continuation byte

To solve this, change you code page to 866

chcp 866

And try to install sqlalchemy once again.

Issue on sqlalchemy: https://bitbucket.org/zzzeek/sqlalchemy/issues/3851/pip-install-error-windows-10

sakurai-youhei commented 7 years ago

+1 for fix; I've seen many (now pysandbox) recently because of code page 932 on my Windows...

C:\Users\sakurai>python --version
Python 3.6.0

C:\Users\sakurai>python -m pip --version
pip 9.0.1 from C:\Python36\lib\site-packages (python 3.6)
xavfernandez commented 7 years ago

Related to #4251

AraHaan commented 7 years ago

I seen this issue also happen with packages that have cython extension modules (aiohttp, yarl) in packages on 3.6 when they did not have wheels for 3.6 on PyPi. Note this bug does not appear when pip installing bdist_wheel's even if there are no c/cython extension modules.

XCanG commented 7 years ago

I want to say some words about that error. In short: Some days ago I can't update aiohttp and today I can't install compleately new package. About aiohttp problem I already write issue here and I solve it: https://github.com/aio-libs/aiohttp/issues/1885 Today I try to talk in IRC on #python channel and I get help and we find source of that error. Here I quote problem and source from chat log:

[22:41:26] <moreati> XCanG: please pastebin what you ran, the output and the errors
[22:59:06] <XCanG> moreati: here I write all about my bug with pip what I know: https://pastebin.com/khmaK0Pu
[22:59:09] <infobob> https://paste.pound-python.org/show/aoxHulQphZhCtlgWk739/ (repasted for XCanG)
[23:00:50] <ChrisWarrick> XCanG: There is some broken file that pip tries to read on your system
[23:02:42] <moreati> XCanG: run pip freeze to get a list of pip-installed packages
[23:03:15] <moreati> XCanG: then run pip uninstall -y <those packages>
[23:03:48] <ChrisWarrick> XCanG: I’d do my favorite way of debugging, print debugging
[23:04:05] <ChrisWarrick> XCanG: open C:\python\python36\lib\site-packages\pip\compat\__init__.py, and before line 75, add: print(s)
[23:04:14] <ChrisWarrick> XCanG: then see/tell us what it prints
[23:09:52] <XCanG> ChrisWarrick print printed that:   Running setup.py install for ujson ... /b'   \xd1\xee\xe7\xe4\xe0\xe5\xf2\xf1\xff \xe1\xe8\xe1\xeb\xe8\xee\xf2\xe5\xea\xe0 build\\temp.win-amd64-3.6\\Release\\./python\\ujson.cp36-win_amd64.lib \xe8 \xee\xe1\xfa\xe5\xea\xf2 build\\temp.win-amd64-3.6\\Release\\./python\\ujson.cp36-win_amd64.exp\r\n'
[23:10:27] <XCanG> and error on 0xd1
[23:11:18] <ChrisWarrick> XCanG: Which language do you speak/is your system set to?
[23:11:36] <XCanG> my system language is russian
[23:12:26] <ChrisWarrick> XCanG: Your problem is “Создаетсябиблиотека”. Report a bug in pip, pastebin traceback, make sure to mention your system is in Russian, and that the string is CP1251.
...
[23:18:22] <ChrisWarrick> XCanG: There seems to be a bug in pip with the handling of Russian (non-latin) path names

My system specs writed in that pastebin url and as Chris say, my system language is Russian and string generated in CP1251. Link to pastebin: https://pastebin.com/khmaK0Pu

If you need anything else, then ask me, I will reply. Hope that helps with solving that error.

pfmoore commented 7 years ago

Fundamentally, I believe the problem is that Visual C can produce output that is in a mix of the ANSI and the OEM codepages when run with stdout directed to a pipe. This means that the output is in practice un-decodable to Unicode.

At the moment, pip fails with a UnicodeError as a result of this. We're working on a patch that allows pip to produce output without failing, although the output will still contain mojibake as it is the data itself that is in error.

XCanG commented 7 years ago

I write fix for me. In /site-packages/pip/compat/__init__.py I add:

from locale import getpreferredencoding

...

if sys.version_info >= (3,):
    def console_to_str(s):
        try:
            ss = []
            l = getpreferredencoding()
            for i in range(len(s)):
                if 255 >= s[i] >= 128:
                    ss.append(s[i:i+1].decode(l))
                else:
                    ss.append(s[i:i+1].decode(sys.__stdout__.encoding))
            s = "".join(ss)
            del ss
            return s
        except UnicodeDecodeError:
            return s.decode('utf_8')

    def native_str(s, replace=False):
        if isinstance(s, bytes):
            return s.decode('utf-8', 'replace' if replace else 'strict')
        return s

So I locale.getpreferredencoding() return my cp1251 Then I check if characters between 128 (\x80) and 255 (\xff) I convert it from my local code page, else I convert by utf-8, then join converted string and return.

Also, what is actually in output (log by print("// LOGGING /", s, end="")):

pip install ujson
Collecting ujson
  Using cached ujson-1.35.tar.gz
// LOGGING / running egg_info
// LOGGING / creating pip-egg-info\ujson.egg-info
// LOGGING / writing pip-egg-info\ujson.egg-info\PKG-INFO
// LOGGING / writing dependency_links to pip-egg-info\ujson.egg-info\dependency_links.txt
// LOGGING / writing top-level names to pip-egg-info\ujson.egg-info\top_level.txt
// LOGGING / writing manifest file 'pip-egg-info\ujson.egg-info\SOURCES.txt'
// LOGGING / warning: manifest_maker: standard file '-c' not found
// LOGGING /
// LOGGING / reading manifest file 'pip-egg-info\ujson.egg-info\SOURCES.txt'
// LOGGING / reading manifest template 'MANIFEST.in'
// LOGGING / writing manifest file 'pip-egg-info\ujson.egg-info\SOURCES.txt'
// LOGGING / Installing collected packages: ujson
  Running setup.py install for ujson ... // LOGGING / running install
-// LOGGING / running build
// LOGGING / running build_ext
// LOGGING / building 'ujson' extension
// LOGGING / creating build
\// LOGGING / creating build\temp.win-amd64-3.6
// LOGGING / creating build\temp.win-amd64-3.6\Release
// LOGGING / creating build\temp.win-amd64-3.6\Release\python
// LOGGING / creating build\temp.win-amd64-3.6\Release\lib
// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I./python -I./lib -Ic:\python\python36\include -Ic:\python\python36\include "-ID:\Prog\Microsoft Visual Studio 14.0\VC\INCLUDE" "-ID:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" /Tc./python/ujson.c /Fobuild\temp.win-amd64-3.6\Release\./python/ujson.obj -D_GNU_SOURCE
// LOGGING / ujson.c
// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I./python -I./lib -Ic:\python\python36\include -Ic:\python\python36\include "-ID:\Prog\Microsoft Visual Studio 14.0\VC\INCLUDE" "-ID:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" /Tc./python/objToJSON.c /Fobuild\temp.win-amd64-3.6\Release\./python/objToJSON.obj -D_GNU_SOURCE
|// LOGGING / objToJSON.c
// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I./python -I./lib -Ic:\python\python36\include -Ic:\python\python36\include "-ID:\Prog\Microsoft Visual Studio 14.0\VC\INCLUDE" "-ID:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" /Tc./python/JSONtoObj.c /Fobuild\temp.win-amd64-3.6\Release\./python/JSONtoObj.obj -D_GNU_SOURCE
// LOGGING / JSONtoObj.c
// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I./python -I./lib -Ic:\python\python36\include -Ic:\python\python36\include "-ID:\Prog\Microsoft Visual Studio 14.0\VC\INCLUDE" "-ID:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" /Tc./lib/ultrajsonenc.c /Fobuild\temp.win-amd64-3.6\Release\./lib/ultrajsonenc.obj -D_GNU_SOURCE
/// LOGGING / ultrajsonenc.c
// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I./python -I./lib -Ic:\python\python36\include -Ic:\python\python36\include "-ID:\Prog\Microsoft Visual Studio 14.0\VC\INCLUDE" "-ID:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" /Tc./lib/ultrajsondec.c /Fobuild\temp.win-amd64-3.6\Release\./lib/ultrajsondec.obj -D_GNU_SOURCE
// LOGGING / ultrajsondec.c
// LOGGING / creating G:\AppData\Local\Temp\pip-build-pgfn501d\ujson\build\lib.win-amd64-3.6
-// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:c:\python\python36\libs /LIBPATH:c:\python\python36\PCbuild\amd64 "/LIBPATH:D:\Prog\Microsoft Visual Studio 14.0\VC\LIB\amd64" "/LIBPATH:D:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.10586.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.10586.0\um\x64" /EXPORT:PyInit_ujson build\temp.win-amd64-3.6\Release\./python/ujson.obj build\temp.win-amd64-3.6\Release\./python/objToJSON.obj build\temp.win-amd64-3.6\Release\./python/JSONtoObj.obj build\temp.win-amd64-3.6\Release\./lib/ultrajsonenc.obj build\temp.win-amd64-3.6\Release\./lib/ultrajsondec.obj /OUT:build\lib.win-amd64-3.6\ujson.cp36-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.6\Release\./python\ujson.cp36-win_amd64.lib
// LOGGING /    Создается библиотека build\temp.win-amd64-3.6\Release\./python\ujson.cp36-win_amd64.lib и объект build\temp.win-amd64-3.6\Release\./python\ujson.cp36-win_amd64.exp
// LOGGING / Создание кода
// LOGGING / Создание кода завершено
\// LOGGING / running install_lib
// LOGGING / copying build\lib.win-amd64-3.6\ujson.cp36-win_amd64.pyd -> c:\python\python36\Lib\site-packages
// LOGGING / running install_egg_info
// LOGGING / running egg_info
// LOGGING / writing ujson.egg-info\PKG-INFO
// LOGGING / writing dependency_links to ujson.egg-info\dependency_links.txt
// LOGGING / writing top-level names to ujson.egg-info\top_level.txt
// LOGGING / warning: manifest_maker: standard file '-c' not found
// LOGGING /
// LOGGING / reading manifest file 'ujson.egg-info\SOURCES.txt'
// LOGGING / reading manifest template 'MANIFEST.in'
// LOGGING / writing manifest file 'ujson.egg-info\SOURCES.txt'
// LOGGING / Copying ujson.egg-info to c:\python\python36\Lib\site-packages\ujson-1.35-py3.6.egg-info
// LOGGING / running install_scripts
// LOGGING / writing list of installed files to 'G:\AppData\Local\Temp\pip-hxlakqjj-record\install-record.txt'
// LOGGING /done
Successfully installed ujson-1.35

C:\Windows\system32>

As you can see, here is 3 lines what contain my local language.

pfmoore commented 7 years ago

This should now be fixed by #4486, which is now available in the development version of pip and will be released in pip 10. I am closing this issue, but if you are still able to reproduce the problem in the development version of pip, please reopen it with a description of now to replicate the issue.

hasufell commented 6 years ago

This should now be fixed by #4486, which is now available in the development version of pip and will be released in pip 10.

So... pip on windows is basically broken since more than half a year without a convenient way for a temporary fix? pip-10 is not released and I don't see any pre-release either.

sakurai-youhei commented 6 years ago

I personally agree with hasufell’s point. Newbies (I guess they’d be students) have been wasting time due to this issue in Japan because of majority of Japanese Windows (cp932). I wish pip were not a blocker but a helper of newbies’ start-up in Python’s ecosystem.

pradyunsg commented 6 years ago

@sakurai-youhei I think there'll be a 10.0 soon. Until then, you could use pip's development version by installing it as follows:

pip install -U git+https://github.com/pypa/pip
AraHaan commented 6 years ago

@pradyunsg you forgot an --upgrade there.

pradyunsg commented 6 years ago

Thanks for catching that @AraHaan!

shimizukawa commented 6 years ago

I've installed 10.0.0.dev0 with bellow command because Windows locks pip.exe and "users" doesn't have git client. It worked.

python -m pip install -U https://github.com/pypa/pip/archive/master.zip
pradyunsg commented 6 years ago

Thanks @shimizukawa!

tbh, I'd not actually put in the amount of thought that I should have in my comment above. Your command is indeed better than what I'd suggested. I might just start suggesting the same soon. :)

uranusjr commented 6 years ago

So a new year has come and there is still no concrete plan for Pip 10. Would it be plausible to have a 9.0.2 to include this fix, please? This is clearly an issue that affects people, and one those affected are highly unlikely to know how to solve.

paulorodriguesxv commented 6 years ago

I have to agree with @uranusjr. I've been watching this issue about one year, it's so long time and some times a think it's kinda windows-users-bullying ?

sakurai-youhei commented 6 years ago

+1 for 9.0.2

pfmoore commented 6 years ago

windows-users-bullying

Not sure what you mean by that. I'm a windows user and a pip dev, and I implemented the fix for this issue. I agree it's high priority to get it released (for some of our users - like most encoding issues, it's user-specific - if it hits you you get it all the time, but if it doesn't you're not clear what the fuss is about). If the pip release process was structured that way, I'd support the idea of a bugfix release, but unfortunately, it's not - we release what's currently on the master branch at any time, and we currently have some new features on master that need a bit of work to complete (PEP 517/518 support in particular, and also the hiding of the internal API, which while appropriate for pip 10, isn't really acceptable for a 9.0.x release).

So while I'd agree that this fix is one I'd be happy to include in a 9.0.2 release, we don't really have an easy way to cut a 9.0.2 release at this time.

One of our goals for a more sustainable pip development process should probably be structuring things so we have a better workflow to support bugfix releases. But we have extremely limited developer resources, and basically no commercial or vendor support, so "big picture" questions like this are hard to find time to address. If anyone wants to raise the issue with whoever provides commercial support for their use of Python/pip, asking them to contribute towards the sustainability of Python's package management infrastructure, that would be a good long-term step to avoiding this type of situation in the future.

paulorodriguesxv commented 6 years ago

Hi @pfmoore. I understand your viewpoint, and you're right about it. About the "windows-users-bullying" I was just kidding, sorry for that. :-)

pfmoore commented 6 years ago

No problem - I just wasn't sure what you meant and whether there was a genuine issue we could address underlying your comment.

XCanG commented 6 years ago

@uranusjr @paulorodriguesxv If you still have this issue, try my code (other fixes not help to me, so I code it by myself). The code showed, probably, above https://github.com/pypa/pip/issues/4110#issuecomment-302927949.

paulorodriguesxv commented 6 years ago

thanks @XCanG

lock[bot] commented 5 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.