menuconfig.py doesn't handle LC_CTYPE identically with Python2 and 3

[pierre-labastie]

Version:

12.4.0

System:

debian 10

Description:

When launching menuconfig.py with Python2, I get: Note: Your environment is configured to use ASCII. To avoid Unicode issues, LC_CTYPE was changed from the C locale to the C.UTF-8 locale. While with Python3, I do not get that.

Locale settings:

LANG=fr_FR.UTF-8
LANGUAGE=
LC_CTYPE="fr_FR.UTF-8"
LC_NUMERIC="fr_FR.UTF-8"
LC_TIME="fr_FR.UTF-8"
LC_COLLATE="fr_FR.UTF-8"
LC_MONETARY="fr_FR.UTF-8"
LC_MESSAGES="fr_FR.UTF-8"
LC_PAPER="fr_FR.UTF-8"
LC_NAME="fr_FR.UTF-8"
LC_ADDRESS="fr_FR.UTF-8"
LC_TELEPHONE="fr_FR.UTF-8"
LC_MEASUREMENT="fr_FR.UTF-8"
LC_IDENTIFICATION="fr_FR.UTF-8"
LC_ALL=

More info:

The documentation for the Python2 locale module tells to call locale.setlocale(locale.LC_ALL,'') to get the user's settings. If tried the following with Python2:

>>> import locale
>>> locale.setlocale(locale.LC_CTYPE)
'C'
>>> locale.setlocale(locale.LC_ALL,'')
'fr_FR.UTF-8'
>>> locale.setlocale(locale.LC_CTYPE)
'fr_FR.UTF-8'

[ulfalizer]

There already is a call to get the settings from the environment, so not sure what might be causing this. It does the same thing as the code you posted at the end.

# Make curses use the locale settings specified in the environment
locale.setlocale(locale.LC_ALL, "")

# Try to fix Unicode issues on systems with bad defaults
if _CONVERT_C_LC_CTYPE_TO_UTF8:
    _convert_c_lc_ctype_to_utf8()

I tried to reproduce the issue with this script, but it works fine on my system (no changing of the locale):

#!/bin/sh

export LANG=sv_SE.UTF-8
export LANGUAGE=
export LC_CTYPE="sv_SE.UTF-8"
export LC_NUMERIC="sv_SE.UTF-8"
export LC_TIME="sv_SE.UTF-8"
export LC_COLLATE="sv_SE.UTF-8"
export LC_MONETARY="sv_SE.UTF-8"
export LC_MESSAGES="sv_SE.UTF-8"
export LC_PAPER="sv_SE.UTF-8"
export LC_NAME="sv_SE.UTF-8"
export LC_ADDRESS="sv_SE.UTF-8"
export LC_TELEPHONE="sv_SE.UTF-8"
export LC_MEASUREMENT="sv_SE.UTF-8"
export LC_IDENTIFICATION="sv_SE.UTF-8"
export LC_ALL=

python2 menuconfig.py

Could you try inserting some print()s into menuconfig.py and comparing the output against your script?

What's your Python 2 version by the way? Mine's 2.7.16, Ubuntu 19.04.

[ulfalizer]

Think I know what might be going on now.

Python 3.7 comes with LC_CTYPE auto-conversion built-in (see PEP 538), so if you're on Python 3.7, then LC_CTYPE will be changed from the C locale to (probably) the C.UTF-8 locale before menuconfig.py even gets a chance to look at it.

That won't happen on Python 2, so menuconfig.py changes it itself instead.

Re. why you end up with LC_CTYPE=C in the first place, it might be getting set to that in the Makefile. Found this in BLFS/Makefile (hunted down which project it was for):

LANG=C
LC_ALL=C

[ulfalizer]

Wonder if I should silence that message from menuconfig.py though. Might be more annoying than helpful.

[pierre-labastie]

What's your Python 2 version by the way? Mine's 2.7.16, Ubuntu 19.04.

Python 2.7.16, too, on debian sid (should become debian 10 within a few days)

Well, sorry, I've not gone through the full code. I've seen that the message I get is in this function:

def _convert_c_lc_ctype_to_utf8():
    # See _CONVERT_C_LC_CTYPE_TO_UTF8

    if _IS_WINDOWS:
        # Windows rarely has issues here, and the PEP 538 implementation avoids
        # changing the locale on it. None of the UTF-8 locales below were
        # supported from some quick testing either. Play it safe.
        return

    def try_set_locale(loc):
        try:
            locale.setlocale(locale.LC_CTYPE, loc)
            return True
        except locale.Error:
            return False

    # Is LC_CTYPE set to the C locale?
    if locale.setlocale(locale.LC_CTYPE, None) == "C":
        # This list was taken from the PEP 538 implementation in the CPython
        # code, in Python/pylifecycle.c
        for loc in "C.UTF-8", "C.utf8", "UTF-8":
            if try_set_locale(loc):
                print("Note: Your environment is configured to use ASCII. To "
                      "avoid Unicode issues, LC_CTYPE was changed from the "
                      "C locale to the {} locale.".format(loc))
                break

so I tried what I've shown in the issue report... I've now put print()'s in the menuconfig.py script, and it shows that LC_CTYPE is always C with Python2, while it is C.UTF-8 with Python3.

Now I see LANG=C is set in the Makefile!!! Sorry for the noise. But still Python3 does not show the same behavior, because it sets the locale to C.UTF-8. I wonder what happens on systems where the C.UTF-8 locale is not installed (such as LFS).

About silencing the message, it might be useful since the conversion is made silently in Python3 anyway... Up to you.

[ulfalizer]

Now I see LANG=C is set in the Makefile!!! Sorry for the noise. But still Python3 does not show the same behavior, because it sets the locale to C.UTF-8. I wonder what happens on systems where the C.UTF-8 locale is not installed (such as LFS).

The PEP 538 code has a built-in list of locales that it tries, and if none of them work, it stays in the C locale. The code in menuconfig.py is a poor man's PEP 538. I copied the list of locales from the C code.

About silencing the message, it might be useful since the conversion is made silently in Python3 anyway... Up to you.

I'll just remove the message. Thinking about it, it might be messy to get rid of it by changing client code, and for Makefiles that set LC_ALL=C (probably common, and might help with other stuff), it'll do the right thing.

[ulfalizer]

Removed now!

No worries about the Makefile. Made me reconsider that message.

[pierre-labastie]

For jhalfs, I'll remove the LC_ALL/LANG settings: they are not needed anymore. And I'll explicitly use python3 to call menuconfig.py, because in LFS, we may have only python3 available.

Thanks for all the pointers.

[ulfalizer]

Sounds good! Python 3 gives better support for Unicode input in text boxes as well, because curses.get_wch() isn't available in Python 2.