venv activate.bat is UTF-8 encoded but uses current console codepage

[151852be-daf3-4b15-9020-b22a5c86e675]

BPO	32409
Nosy	@pfmoore, @vsajip, @vstinner, @tjguk, @ezio-melotti, @zware, @eryksun, @zooba, @pablogsal, @miss-islington, @samstagern
PRs	python/cpython#5757 python/cpython#5765 python/cpython#5765 python/cpython#5766 python/cpython#10295 python/cpython#10377 python/cpython#10403 python/cpython#10405 python/cpython#8321

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/zooba' closed_at = created_at = labels = ['type-bug', '3.8', 'OS-windows', '3.7', 'library', 'expert-unicode'] title = 'venv activate.bat is UTF-8 encoded but uses current console codepage' updated_at = user = 'https://bugs.python.org/Jac0' ``` bugs.python.org fields: ```python activity = actor = 'steve.dower' assignee = 'steve.dower' closed = True closed_date = closer = 'steve.dower' components = ['Library (Lib)', 'Unicode', 'Windows'] creation = creator = 'Jac0' dependencies = [] files = [] hgrepos = [] issue_num = 32409 keywords = ['patch'] message_count = 18.0 messages = ['308931', '308934', '312356', '312389', '312390', '312393', '329141', '329142', '329425', '329431', '329440', '329447', '329450', '329451', '329464', '329481', '332324', '348505'] nosy_count = 12.0 nosy_names = ['paul.moore', 'vinay.sajip', 'vstinner', 'tim.golden', 'ezio.melotti', 'zach.ware', 'eryksun', 'steve.dower', 'pablogsal', 'Jac0', 'miss-islington', 'Martin Bijl-Schwab'] pr_nums = ['5757', '5765', '5765', '5766', '10295', '10377', '10403', '10405', '8321'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue32409' versions = ['Python 3.6', 'Python 3.7', 'Python 3.8'] ```

[151852be-daf3-4b15-9020-b22a5c86e675]

Let's say I have a folder c:\test-ä in Windows

Now if I run: py -m venv env and activate: env\scripts\activate and check: where python

the result is incorrectly just: C:\Users\Username\AppData\Local\Programs\Python\Python36\python.exe

If I run: path the result is: PATH=C:\test-ä\env\Scripts;...

So clearly the encoding is broken for the folder name.

I can fix this by changing activate.bat character encoding to OEM-US and then replacing "test-├ż" by "test-ä".

If I now activate and run: where python the result is (as should be): C:\test-ä\env\Scripts\python.exe C:\Users\Username\AppData\Local\Programs\Python\Python36\python.exe

By running: path I get: PATH=C:\test-ä\env\Scripts;...

So looks good here as well.

I suggest that what ever is creating activate.bat file, is using incorrect character encoding for the creation of the file. If this is somehow platform specific, there could be a guide in the venv documentation about how to fix this.

[eryksun]

The CMD shell decodes batch scripts using the attached console's output codepage, which defaults to OEM. OTOH, venv writes the replacement values for the template activate.bat as UTF-8 (codepage 65001), which is correct and should not be downgraded to OEM.

Instead, the batch script could temporarily switch the console to codepage 65001. Then restore the previous codepage at the end. For example:

@echo off
for /f "tokens=2 delims=:" %%a in ('"%SystemRoot%\System32\chcp.com"') do (
    set "CODEPAGE=%%a"
)
"%SystemRoot%\System32\chcp.com" 65001 > nul

[rest of script]

"%SystemRoot%\System32\chcp.com" %CODEPAGE% > nul
set "CODEPAGE="
:END

[zooba]

Eryk's solution seems to be best, so I'll add that.

[zooba]

New changeset 6240917b773b52f8883387b9e3a5f327a4372068 by Steve Dower in branch 'master': bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5757) https://github.com/python/cpython/commit/6240917b773b52f8883387b9e3a5f327a4372068

[zooba]

New changeset a3d6c1b23b8a49b5003fdbd115d3598fe3d4c4bf by Steve Dower (Miss Islington (bot)) in branch '3.7': bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5765) https://github.com/python/cpython/commit/a3d6c1b23b8a49b5003fdbd115d3598fe3d4c4bf

[zooba]

New changeset 8e149ff481acbb3889c825b8bf7b10aa191f09a7 by Steve Dower (Miss Islington (bot)) in branch '3.6': bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5766) https://github.com/python/cpython/commit/8e149ff481acbb3889c825b8bf7b10aa191f09a7

[7e995f47-e667-4a9c-9094-993e6dd4feb1]

it does not work as expected on swiss german (and likely other internationalised) windows systems. See https://bugs.python.org/issue32409

[7e995f47-e667-4a9c-9094-993e6dd4feb1]

I meant https://bugs.python.org/issue35148

[vsajip]

New changeset c64583b6d3e8516a8cd2b5f84fc1e300bfac2206 by Vinay Sajip (samstagern) in branch 'master': bpo-32409: Fix regression in activate.bat on international Windows (GH-10295) https://github.com/python/cpython/commit/c64583b6d3e8516a8cd2b5f84fc1e300bfac2206

[vsajip]

New changeset 881e273c795f2f5154b8afebfa299f0e830f3712 by Vinay Sajip (Miss Islington (bot)) in branch '3.7': bpo-32409: Fix regression in activate.bat on international Windows (GH-10295) (GH-10377) https://github.com/python/cpython/commit/881e273c795f2f5154b8afebfa299f0e830f3712

[pablogsal]

This commit broke a lot of the Windows buildbots:

https://buildbot.python.org/all/#/builders/32/builds/1707 https://buildbot.python.org/all/#/builders/113/builds/733 https://buildbot.python.org/all/#/builders/40/builds/1137 https://buildbot.python.org/all/#/builders/130/builds/420 https://buildbot.python.org/all/#/builders/3/builds/1634

Can someone work on a patch soon? If not we need to revert the commits in all the branches that are failed to build :(

[7e995f47-e667-4a9c-9094-993e6dd4feb1]

I will have a look.

[pablogsal]

New changeset 6843ffe4533e9f2cde036296fd932fef6f059687 by Pablo Galindo in branch 'master': Revert "bpo-32409: Fix regression in activate.bat on international Windows (GH-10295)" (GH-10403) https://github.com/python/cpython/commit/6843ffe4533e9f2cde036296fd932fef6f059687

[miss-islington]

New changeset 3ba5e253de1a52e9692aa645c95072705f08b7bb by Miss Islington (bot) in branch '3.7': Revert "bpo-32409: Fix regression in activate.bat on international Windows (GH-10295)" (GH-10403) https://github.com/python/cpython/commit/3ba5e253de1a52e9692aa645c95072705f08b7bb

[vstinner]

Pablo Galindo reverted the change because it broke Windows buildbots, and we have a policy to revert a change breaking buildbots if the regression cannot be fixed "quickly" event: https://pythondev.readthedocs.io/ci.html#revert-on-fail

Is someone working on investigating the bug? Do you need help to reproduce the bug?

Copy of the test_venv error:

\====================================================================== ERROR: test_unicode_in_batch_file (test.test_venv.BasicTest) ----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\buildarea\3.x.bolen-windows8\build\lib\test\test_venv.py", line 302, in test_unicode_in_batch_file
    out, err = check_output(
  File "D:\buildarea\3.x.bolen-windows8\build\lib\test\test_venv.py", line 37, in check_output
    raise subprocess.CalledProcessError(
TypeError: __init__() takes from 3 to 5 positional arguments but 6 were given

https://buildbot.python.org/all/#/builders/32/builds/1707

[zooba]

That error is a bug in the test, but it only shows up on an error path anyway. Without removing the extra None we don't get to see the actual error output.

I can't look into this over the next week or two, but a quick glance at the original PR looks like a lot of quotes are missing around executable paths, so maybe it was that?

[vsajip]

See also bpo-35558.

[zooba]

This appears to be completely resolved now, but if it's not, please ping with the details and we'll reopen.