[Windows] os.device_encoding(fd) should support any console fd

BPO	43421
Nosy	@pfmoore, @tjguk, @zware, @eryksun, @zooba

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['3.8', '3.9', 'extension-modules', 'interpreter-core', 'type-bug', 'expert-IO', '3.10', 'OS-windows'] title = 'os.device_encoding(fd) should support any console fd in Windows' updated_at = user = 'https://github.com/eryksun' ``` bugs.python.org fields: ```python activity = actor = 'eryksun' assignee = 'none' closed = False closed_date = None closer = None components = ['Extension Modules', 'Interpreter Core', 'Windows', 'IO'] creation = creator = 'eryksun' dependencies = [] files = [] hgrepos = [] issue_num = 43421 keywords = [] message_count = 1.0 messages = ['388201'] nosy_count = 5.0 nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower'] pr_nums = [] priority = 'normal' resolution = None stage = None status = 'open' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue43421' versions = ['Python 3.8', 'Python 3.9', 'Python 3.10'] ```

In Windows, os.device_encoding() is hard coded to map file descriptor 0 and descriptors 1 and 2 respectively to the console's input and output code page if isatty(fd) is true. But isatty() is true for any character device, such as "NUL". Also any fd might be a console, by way of dup(), dup2() -- or open() with the device names "CON", "CONIN$", and "CONOUT$".

The correct device encoding of a console file is needed for use with os.read() and os.write(). It's also necessary for io.TextIOWrapper() if PYTHONLEGACYWINDOWSSTDIO is set.

_Py_device_encoding() in Python/fileutils.c should use _get_osfhandle() to get the OS handle, and, if it's a character-device file, determine the code page to return, if any, depending on whether it's an input or output console file. For example:

PyObject *
_Py_device_encoding(int fd)
{
#if defined(MS_WINDOWS)
    HANDLE handle;
    DWORD temp;
    UINT cp = 0;

    _Py_BEGIN_SUPPRESS_IPH
        handle = (HANDLE)_get_osfhandle(fd);
    _Py_END_SUPPRESS_IPH
    if (handle == INVALID_HANDLE_VALUE ||
            GetFileType(handle) != FILE_TYPE_CHAR)
        Py_RETURN_NONE;

    Py_BEGIN_ALLOW_THREADS
    /* GetConsoleMode requires a console handle. */
    if (!GetConsoleMode(handle, &temp)) {
        /* Assume access denied implies output. */
        if (GetLastError() == ERROR_ACCESS_DENIED)
            cp = GetConsoleOutputCP();
    } else {
        if (GetNumberOfConsoleInputEvents(handle, &temp)) {
            cp = GetConsoleCP();
        } else {
            cp = GetConsoleOutputCP();
        }
    }
    Py_END_ALLOW_THREADS

    if (cp == CP_UTF8) {
        return PyUnicode_FromString("UTF-8");
    } else if (cp != 0) {
        return PyUnicode_FromFormat("cp%u", (unsigned int)cp);
    } else {
        Py_RETURN_NONE;
    }
#else
    if (isatty(fd)) {
        return _Py_GetLocaleEncodingObject();
    } else {
        Py_RETURN_NONE;
    }
#endif  /* defined(MS_WINDOWS) */
}

python / cpython

[Windows] os.device_encoding(fd) should support any console fd #87587