Interaction test is broken on Python 2.7 if LANG=C

[mnencia]

When you run the test suite on Python 2.7 using LANG=C it fails with an error.

$ LANG=C tox -e py27
GLOB sdist-make: /Users/mnencia/prj/barman/argh/setup.py
py27 inst-nodeps: /Users/mnencia/prj/barman/argh/.tox/dist/argh-0.26.1.zip
py27 runtests: PYTHONHASHSEED='1730616988'
py27 runtests: commands[0] | py.test
============================= test session starts ==============================
platform darwin -- Python 2.7.9 -- py-1.4.26 -- pytest-2.6.4
plugins: cov, xdist
collected 80 items

test/test_assembling.py .......ss...ss..
test/test_compat.py .
test/test_decorators.py ......
test/test_dispatching.py ..
test/test_integration.py x.......xxx..............................s....
test/test_interaction.py ..F
test/test_regressions.py ......

=================================== FAILURES ===================================
________________________________ test_encoding _________________________________

    def test_encoding():
        "Unicode is accepted as prompt message"
        raw_input_mock = mock.MagicMock()

        argh.io._input = raw_input_mock

        msg = 'привет'
        if sys.version_info <= (3,0):
            msg = msg.decode('utf-8')

>       argh.confirm(msg)

test/test_interaction.py:67:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
argh/interaction.py:72: in confirm
    choice = safe_input(prompt)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

prompt = '\u043f\u0440\u0438\u0432\u0435\u0442? (y/n)'

    def safe_input(prompt):
        """
        Prompts user for input. Correctly handles prompt message encoding.
        """

        if sys.version_info < (3,0):
            if isinstance(prompt, compat.text_type):
                # Python 2.x: unicode →  bytes
                encoding = locale.getpreferredencoding() or 'utf-8'
>               prompt = prompt.encode(encoding)
E               UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-5: ordinal not in range(128)

argh/io.py:41: UnicodeEncodeError
========== 1 failed, 70 passed, 5 skipped, 4 xfailed in 0.30 seconds ===========
ERROR: InvocationError: '/Users/mnencia/prj/barman/argh/.tox/py27/bin/py.test'
___________________________________ summary ____________________________________
ERROR:   py27: commands failed

It is important because Debian build daemons always run with LANG=C and fails to build the package.

As a workaround the issue I marked the test as xfail on that particular combination

@pytest.mark.xfail(sys.version_info <= (3,0) and os.environ['LANG'] == 'C', reason='It fails on buildd')
def test_encoding():

[joequery]

Hey there!

I was unable to replicate this with Ubuntu 14.04, Python 2.7.10.

Side note, I thought

platform darwin -- Python 2.7.9 -- py-1.4.26 -- pytest-2.6.4

Implied this test was performed on OS X, but your concerns are about Debian

[mnencia]

I tested it on both Debian and OSX, and the issue was present on both. However it was long time ago, Tomorrow I'll test it again and I'll give you more details.

[mnencia]

I can confirm it. I've just obtained the same error on a fresh clone on a Debian Jessie server.

mnencia@spark:~/tmp/argh$ LANG=C tox -e py27
GLOB sdist-make: /home/mnencia/tmp/argh/setup.py
py27 create: /home/mnencia/tmp/argh/.tox/py27
py27 installdeps: coverage, mock, pytest, pytest-cov
py27 inst: /home/mnencia/tmp/argh/.tox/dist/argh-0.26.1.zip
py27 installed: argh==0.26.1,coverage==3.7.1,funcsigs==0.4,mock==1.3.0,pbr==1.6.0,py==1.4.30,pytest==2.7.2,pytest-cov==2.1.0,six==1.9.0,wheel==0.24.0
py27 runtests: PYTHONHASHSEED='4223145432'
py27 runtests: commands[0] | py.test
================================================================================ test session starts ================================================================================
platform linux2 -- Python 2.7.9 -- py-1.4.30 -- pytest-2.7.2
rootdir: /home/mnencia/tmp/argh, inifile:
plugins: cov
collected 80 items

test/test_assembling.py .......ss...ss..
test/test_compat.py .
test/test_decorators.py ......
test/test_dispatching.py ..
test/test_integration.py x.......xxx..............................s....
test/test_interaction.py ..F
test/test_regressions.py ......

===================================================================================== FAILURES ======================================================================================
___________________________________________________________________________________ test_encoding ___________________________________________________________________________________

    def test_encoding():
        "Unicode is accepted as prompt message"
        raw_input_mock = mock.MagicMock()

        argh.io._input = raw_input_mock

        msg = 'привет'
        if sys.version_info <= (3,0):
            msg = msg.decode('utf-8')

>       argh.confirm(msg)

test/test_interaction.py:67:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
argh/interaction.py:72: in confirm
    choice = safe_input(prompt)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

prompt = '\u043f\u0440\u0438\u0432\u0435\u0442? (y/n)'

    def safe_input(prompt):
        """
        Prompts user for input. Correctly handles prompt message encoding.
        """

        if sys.version_info < (3,0):
            if isinstance(prompt, compat.text_type):
                # Python 2.x: unicode →  bytes
                encoding = locale.getpreferredencoding() or 'utf-8'
>               prompt = prompt.encode(encoding)
E               UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-5: ordinal not in range(128)

argh/io.py:41: UnicodeEncodeError
============================================================= 1 failed, 70 passed, 5 skipped, 4 xfailed in 0.47 seconds =============================================================
ERROR: InvocationError: '/home/mnencia/tmp/argh/.tox/py27/bin/py.test'
______________________________________________________________________________________ summary ______________________________________________________________________________________
ERROR:   py27: commands failed

python 2.7.9 has been installed through pyenv

packages versione are:

$ pip freeze --local
blessings==1.6
bpython==0.14.2
curtsies==0.2.3
decorator==4.0.2
greenlet==0.4.9
ipython==4.0.0
ipython-genutils==0.1.0
path.py==8.1
pexpect==3.3
pickleshare==0.5
pluggy==0.3.0
psycopg2==2.6.1
py==1.4.30
Pygments==2.0.2
requests==2.7.0
simplegeneric==0.8.1
six==1.9.0
tox==2.1.1
traitlets==4.0.0
virtualenv==13.1.2
wcwidth==0.1.4

[joequery]

Cool, thanks for taking the time to do this. I'll try to reproduce using 2.7.9

[anthraxx]

I can confirm this issue with python 2.7.10 (works fine with python3), any progress on this issue?

[kpcyrd]

I've reproduced this with python 2.7.10 on linux.

Forcing utf-8 fixes the test, but I'm not sure this is the best way to go.

diff --git a/argh/io.py b/argh/io.py
index 35e9006..503293b 100644
--- a/argh/io.py
+++ b/argh/io.py
@@ -38,6 +38,8 @@ def safe_input(prompt):
         if isinstance(prompt, compat.text_type):
             # Python 2.x: unicode →  bytes
             encoding = locale.getpreferredencoding() or 'utf-8'
+            if encoding == 'ANSI_X3.4-1968':
+                encoding = 'utf-8'
             prompt = prompt.encode(encoding)
     else:
         if not isinstance(prompt, compat.text_type):

[jwilk]

@mnencia's workaround (sys.version_info <= (3,0) and os.environ['LANG'] == 'C'...) is incorrect. Setting LANG=C is, in general, neither sufficient nor necessary to force the C locale.

@kpcyrd, you can't assume that locale.getpreferredencoding() returns exactly 'ANSI_X3.4-1968' when the encoding is ASCII. For example, on FreeBSD the result is 'US-ASCII'.

[jwilk]

@kpcyrd, and then, falling back from ASCII to UTF-8 doesn't necessarily help, because this test fails also in some non-ASCII locales, such as en_US.iso88591.

[neithere]

Closing the issue as Python 2.7 is already long dead.