Closed Goblenus closed 7 years ago
+1 for fix; I've seen many (now pysandbox) recently because of code page 932 on my Windows...
C:\Users\sakurai>python --version
Python 3.6.0
C:\Users\sakurai>python -m pip --version
pip 9.0.1 from C:\Python36\lib\site-packages (python 3.6)
Related to #4251
I seen this issue also happen with packages that have cython extension modules (aiohttp, yarl) in packages on 3.6 when they did not have wheels for 3.6 on PyPi. Note this bug does not appear when pip installing bdist_wheel's even if there are no c/cython extension modules.
I want to say some words about that error. In short: Some days ago I can't update aiohttp and today I can't install compleately new package. About aiohttp problem I already write issue here and I solve it: https://github.com/aio-libs/aiohttp/issues/1885 Today I try to talk in IRC on #python channel and I get help and we find source of that error. Here I quote problem and source from chat log:
[22:41:26] <moreati> XCanG: please pastebin what you ran, the output and the errors
[22:59:06] <XCanG> moreati: here I write all about my bug with pip what I know: https://pastebin.com/khmaK0Pu
[22:59:09] <infobob> https://paste.pound-python.org/show/aoxHulQphZhCtlgWk739/ (repasted for XCanG)
[23:00:50] <ChrisWarrick> XCanG: There is some broken file that pip tries to read on your system
[23:02:42] <moreati> XCanG: run pip freeze to get a list of pip-installed packages
[23:03:15] <moreati> XCanG: then run pip uninstall -y <those packages>
[23:03:48] <ChrisWarrick> XCanG: I’d do my favorite way of debugging, print debugging
[23:04:05] <ChrisWarrick> XCanG: open C:\python\python36\lib\site-packages\pip\compat\__init__.py, and before line 75, add: print(s)
[23:04:14] <ChrisWarrick> XCanG: then see/tell us what it prints
[23:09:52] <XCanG> ChrisWarrick print printed that: Running setup.py install for ujson ... /b' \xd1\xee\xe7\xe4\xe0\xe5\xf2\xf1\xff \xe1\xe8\xe1\xeb\xe8\xee\xf2\xe5\xea\xe0 build\\temp.win-amd64-3.6\\Release\\./python\\ujson.cp36-win_amd64.lib \xe8 \xee\xe1\xfa\xe5\xea\xf2 build\\temp.win-amd64-3.6\\Release\\./python\\ujson.cp36-win_amd64.exp\r\n'
[23:10:27] <XCanG> and error on 0xd1
[23:11:18] <ChrisWarrick> XCanG: Which language do you speak/is your system set to?
[23:11:36] <XCanG> my system language is russian
[23:12:26] <ChrisWarrick> XCanG: Your problem is “Создаетсябиблиотека”. Report a bug in pip, pastebin traceback, make sure to mention your system is in Russian, and that the string is CP1251.
...
[23:18:22] <ChrisWarrick> XCanG: There seems to be a bug in pip with the handling of Russian (non-latin) path names
My system specs writed in that pastebin url and as Chris say, my system language is Russian and string generated in CP1251. Link to pastebin: https://pastebin.com/khmaK0Pu
If you need anything else, then ask me, I will reply. Hope that helps with solving that error.
Fundamentally, I believe the problem is that Visual C can produce output that is in a mix of the ANSI and the OEM codepages when run with stdout directed to a pipe. This means that the output is in practice un-decodable to Unicode.
At the moment, pip fails with a UnicodeError as a result of this. We're working on a patch that allows pip to produce output without failing, although the output will still contain mojibake as it is the data itself that is in error.
I write fix for me.
In /site-packages/pip/compat/__init__.py
I add:
from locale import getpreferredencoding
...
if sys.version_info >= (3,):
def console_to_str(s):
try:
ss = []
l = getpreferredencoding()
for i in range(len(s)):
if 255 >= s[i] >= 128:
ss.append(s[i:i+1].decode(l))
else:
ss.append(s[i:i+1].decode(sys.__stdout__.encoding))
s = "".join(ss)
del ss
return s
except UnicodeDecodeError:
return s.decode('utf_8')
def native_str(s, replace=False):
if isinstance(s, bytes):
return s.decode('utf-8', 'replace' if replace else 'strict')
return s
So I locale.getpreferredencoding()
return my cp1251
Then I check if characters between 128 (\x80
) and 255 (\xff
) I convert it from my local code page, else I convert by utf-8, then join converted string and return.
Also, what is actually in output (log by print("// LOGGING /", s, end="")
):
pip install ujson
Collecting ujson
Using cached ujson-1.35.tar.gz
// LOGGING / running egg_info
// LOGGING / creating pip-egg-info\ujson.egg-info
// LOGGING / writing pip-egg-info\ujson.egg-info\PKG-INFO
// LOGGING / writing dependency_links to pip-egg-info\ujson.egg-info\dependency_links.txt
// LOGGING / writing top-level names to pip-egg-info\ujson.egg-info\top_level.txt
// LOGGING / writing manifest file 'pip-egg-info\ujson.egg-info\SOURCES.txt'
// LOGGING / warning: manifest_maker: standard file '-c' not found
// LOGGING /
// LOGGING / reading manifest file 'pip-egg-info\ujson.egg-info\SOURCES.txt'
// LOGGING / reading manifest template 'MANIFEST.in'
// LOGGING / writing manifest file 'pip-egg-info\ujson.egg-info\SOURCES.txt'
// LOGGING / Installing collected packages: ujson
Running setup.py install for ujson ... // LOGGING / running install
-// LOGGING / running build
// LOGGING / running build_ext
// LOGGING / building 'ujson' extension
// LOGGING / creating build
\// LOGGING / creating build\temp.win-amd64-3.6
// LOGGING / creating build\temp.win-amd64-3.6\Release
// LOGGING / creating build\temp.win-amd64-3.6\Release\python
// LOGGING / creating build\temp.win-amd64-3.6\Release\lib
// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I./python -I./lib -Ic:\python\python36\include -Ic:\python\python36\include "-ID:\Prog\Microsoft Visual Studio 14.0\VC\INCLUDE" "-ID:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" /Tc./python/ujson.c /Fobuild\temp.win-amd64-3.6\Release\./python/ujson.obj -D_GNU_SOURCE
// LOGGING / ujson.c
// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I./python -I./lib -Ic:\python\python36\include -Ic:\python\python36\include "-ID:\Prog\Microsoft Visual Studio 14.0\VC\INCLUDE" "-ID:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" /Tc./python/objToJSON.c /Fobuild\temp.win-amd64-3.6\Release\./python/objToJSON.obj -D_GNU_SOURCE
|// LOGGING / objToJSON.c
// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I./python -I./lib -Ic:\python\python36\include -Ic:\python\python36\include "-ID:\Prog\Microsoft Visual Studio 14.0\VC\INCLUDE" "-ID:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" /Tc./python/JSONtoObj.c /Fobuild\temp.win-amd64-3.6\Release\./python/JSONtoObj.obj -D_GNU_SOURCE
// LOGGING / JSONtoObj.c
// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I./python -I./lib -Ic:\python\python36\include -Ic:\python\python36\include "-ID:\Prog\Microsoft Visual Studio 14.0\VC\INCLUDE" "-ID:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" /Tc./lib/ultrajsonenc.c /Fobuild\temp.win-amd64-3.6\Release\./lib/ultrajsonenc.obj -D_GNU_SOURCE
/// LOGGING / ultrajsonenc.c
// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I./python -I./lib -Ic:\python\python36\include -Ic:\python\python36\include "-ID:\Prog\Microsoft Visual Studio 14.0\VC\INCLUDE" "-ID:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10586.0\winrt" /Tc./lib/ultrajsondec.c /Fobuild\temp.win-amd64-3.6\Release\./lib/ultrajsondec.obj -D_GNU_SOURCE
// LOGGING / ultrajsondec.c
// LOGGING / creating G:\AppData\Local\Temp\pip-build-pgfn501d\ujson\build\lib.win-amd64-3.6
-// LOGGING / D:\Prog\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:c:\python\python36\libs /LIBPATH:c:\python\python36\PCbuild\amd64 "/LIBPATH:D:\Prog\Microsoft Visual Studio 14.0\VC\LIB\amd64" "/LIBPATH:D:\Prog\Microsoft Visual Studio 14.0\VC\ATLMFC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.10586.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.10586.0\um\x64" /EXPORT:PyInit_ujson build\temp.win-amd64-3.6\Release\./python/ujson.obj build\temp.win-amd64-3.6\Release\./python/objToJSON.obj build\temp.win-amd64-3.6\Release\./python/JSONtoObj.obj build\temp.win-amd64-3.6\Release\./lib/ultrajsonenc.obj build\temp.win-amd64-3.6\Release\./lib/ultrajsondec.obj /OUT:build\lib.win-amd64-3.6\ujson.cp36-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.6\Release\./python\ujson.cp36-win_amd64.lib
// LOGGING / Создается библиотека build\temp.win-amd64-3.6\Release\./python\ujson.cp36-win_amd64.lib и объект build\temp.win-amd64-3.6\Release\./python\ujson.cp36-win_amd64.exp
// LOGGING / Создание кода
// LOGGING / Создание кода завершено
\// LOGGING / running install_lib
// LOGGING / copying build\lib.win-amd64-3.6\ujson.cp36-win_amd64.pyd -> c:\python\python36\Lib\site-packages
// LOGGING / running install_egg_info
// LOGGING / running egg_info
// LOGGING / writing ujson.egg-info\PKG-INFO
// LOGGING / writing dependency_links to ujson.egg-info\dependency_links.txt
// LOGGING / writing top-level names to ujson.egg-info\top_level.txt
// LOGGING / warning: manifest_maker: standard file '-c' not found
// LOGGING /
// LOGGING / reading manifest file 'ujson.egg-info\SOURCES.txt'
// LOGGING / reading manifest template 'MANIFEST.in'
// LOGGING / writing manifest file 'ujson.egg-info\SOURCES.txt'
// LOGGING / Copying ujson.egg-info to c:\python\python36\Lib\site-packages\ujson-1.35-py3.6.egg-info
// LOGGING / running install_scripts
// LOGGING / writing list of installed files to 'G:\AppData\Local\Temp\pip-hxlakqjj-record\install-record.txt'
// LOGGING /done
Successfully installed ujson-1.35
C:\Windows\system32>
As you can see, here is 3 lines what contain my local language.
This should now be fixed by #4486, which is now available in the development version of pip and will be released in pip 10. I am closing this issue, but if you are still able to reproduce the problem in the development version of pip, please reopen it with a description of now to replicate the issue.
This should now be fixed by #4486, which is now available in the development version of pip and will be released in pip 10.
So... pip on windows is basically broken since more than half a year without a convenient way for a temporary fix? pip-10 is not released and I don't see any pre-release either.
I personally agree with hasufell’s point. Newbies (I guess they’d be students) have been wasting time due to this issue in Japan because of majority of Japanese Windows (cp932). I wish pip were not a blocker but a helper of newbies’ start-up in Python’s ecosystem.
@sakurai-youhei I think there'll be a 10.0 soon. Until then, you could use pip's development version by installing it as follows:
pip install -U git+https://github.com/pypa/pip
@pradyunsg you forgot an --upgrade
there.
Thanks for catching that @AraHaan!
I've installed 10.0.0.dev0 with bellow command because Windows locks pip.exe and "users" doesn't have git client. It worked.
python -m pip install -U https://github.com/pypa/pip/archive/master.zip
Thanks @shimizukawa!
tbh, I'd not actually put in the amount of thought that I should have in my comment above. Your command is indeed better than what I'd suggested. I might just start suggesting the same soon. :)
So a new year has come and there is still no concrete plan for Pip 10. Would it be plausible to have a 9.0.2 to include this fix, please? This is clearly an issue that affects people, and one those affected are highly unlikely to know how to solve.
I have to agree with @uranusjr. I've been watching this issue about one year, it's so long time and some times a think it's kinda windows-users-bullying ?
+1 for 9.0.2
windows-users-bullying
Not sure what you mean by that. I'm a windows user and a pip dev, and I implemented the fix for this issue. I agree it's high priority to get it released (for some of our users - like most encoding issues, it's user-specific - if it hits you you get it all the time, but if it doesn't you're not clear what the fuss is about). If the pip release process was structured that way, I'd support the idea of a bugfix release, but unfortunately, it's not - we release what's currently on the master branch at any time, and we currently have some new features on master that need a bit of work to complete (PEP 517/518 support in particular, and also the hiding of the internal API, which while appropriate for pip 10, isn't really acceptable for a 9.0.x release).
So while I'd agree that this fix is one I'd be happy to include in a 9.0.2 release, we don't really have an easy way to cut a 9.0.2 release at this time.
One of our goals for a more sustainable pip development process should probably be structuring things so we have a better workflow to support bugfix releases. But we have extremely limited developer resources, and basically no commercial or vendor support, so "big picture" questions like this are hard to find time to address. If anyone wants to raise the issue with whoever provides commercial support for their use of Python/pip, asking them to contribute towards the sustainability of Python's package management infrastructure, that would be a good long-term step to avoiding this type of situation in the future.
Hi @pfmoore. I understand your viewpoint, and you're right about it. About the "windows-users-bullying" I was just kidding, sorry for that. :-)
No problem - I just wasn't sure what you meant and whether there was a genuine issue we could address underlying your comment.
@uranusjr @paulorodriguesxv If you still have this issue, try my code (other fixes not help to me, so I code it by myself). The code showed, probably, above https://github.com/pypa/pip/issues/4110#issuecomment-302927949.
thanks @XCanG
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
I have changed code page in my cmd to 65001 (it's utf-8), and while installing sqlalchemy by pip:
I obtained next error:
To solve this, change you code page to 866
And try to install sqlalchemy once again.
Issue on sqlalchemy: https://bitbucket.org/zzzeek/sqlalchemy/issues/3851/pip-install-error-windows-10