patch_stdout

[iwasz]

Hi, I have problems with an ./asyncio-prompt.py example. Running it under Ubuntu 18.10, Python 3.6.8 source taken from git master branch (today) I encountered :

./asyncio-prompt.py Traceback (most recent call last): File "./asyncio-prompt.py", line 64, in main() File "./asyncio-prompt.py", line 57, in main loop.run_until_complete(shell_task) File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete return future.result() File "./asyncio-prompt.py", line 41, in interactive_shell session = PromptSession('Say something: ') File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/shortcuts/prompt.py", line 433, in init File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/shortcuts/prompt.py", line 671, in _create_application File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/application/application.py", line 260, in init File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/application/current.py", line 61, in output File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/output/defaults.py", line 29, in create_output AttributeError: 'StdoutProxy' object has no attribute 'isatty'

After adding isatty method (my method returns always True), the error has changed to:

./asyncio-prompt.py Counter: 0 Traceback (most recent call last): File "./asyncio-prompt.py", line 64, in main() File "./asyncio-prompt.py", line 57, in main loop.run_until_complete(shell_task) File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete return future.result() File "./asyncio-prompt.py", line 46, in interactive_shell result = await session.prompt_async () File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/shortcuts/prompt.py", line 1030, in prompt_async File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/application/application.py", line 738, in run_async File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/application/application.py", line 723, in _run_async2 File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/application/application.py", line 664, in _run_async File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/application/application.py", line 878, in _request_absolute_cursor_position File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/renderer.py", line 443, in request_absolute_cursor_position File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/renderer.py", line 435, in do_cpr File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/output/vt100.py", line 663, in ask_for_cpr File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/output/vt100.py", line 634, in flush File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/patch_stdout.py", line 141, in write File "/usr/local/lib/python3.6/dist-packages/prompt_toolkit-3.0.0-py3.6.egg/prompt_toolkit/patch_stdout.py", line 121, in _write TypeError: a bytes-like object is required, not 'str'

The only thing I came up to with this then, was to remove "with patch_stdout():" from the example whatsoever.

Best regards.

[lillian27]

iwasz,

Not a fix but more information about the issue.

patch_stdout.py is missing isatty as you observed, and should probably just pass through that property from the original stdout:

def isatty(self) -> bool:
        return self.original_stdout.isatty()

I believe the bytes-alike vs str issue is due to how the StdoutProxy class now works. The TypeErrors are the result of PT trying to get a cursor position report from the terminal by writing '\x1b[6n' to what it thinks is stdout .

output/vt100.py

    def ask_for_cpr(self) -> None:
        """
        Asks for a cursor position report (CPR).
        """
        self.write_raw('\x1b[6n')
        self.flush()

So Vt100Output does a raw write to its internal buffer - fine so far - but when it calls flush() this happens:

output/vt100.py

            if self.write_binary:
                if hasattr(self.stdout, 'buffer'):
                    out = self.stdout.buffer  # Py3.
                else:
                    out = self.stdout
                out.write(data.encode(self.stdout.encoding or 'utf-8', 'replace'))
            else:
                self.stdout.write(data)

The first 'if' is True, the second False, so the buffer gets written to self.stdout which is a StdoutProxy object. The proxy then tries to split out newlines (to avoid excessive write() calls, see stdout_proxy.py:113), and expecting a 'str' gets a bytes-like instead, raising the TypeError. If the data variable at line 121 is examined it does hold the control code for cursor position.

StdoutProxy understands about writing binary vs not but only when invoked:

    def __init__(self, raw: bool = False,
                 original_stdout: Optional[TextIO] = None) -> None:
        ...
        self._raw = raw

However the _raw attribute is not actually used anywhere so it looks like the proxy needs to be extended to enable raw/cooked mode switching as needed. This is as far down the rabbit hole as I went. I don't know how Jonathan might want to fix this since 3.0 is in-progress so I haven't tried to fix it myself. I hope my analysis isn't too badly incorrect ;-)

[iwasz]

Wow, very through analysis. I added my stub isatty only to check if it's the only problem that was, or not. By any means I wanted to propose this as a fix :D Thanks.

On Wed, Jul 3, 2019 at 11:14 PM lillian27 notifications@github.com wrote:

iwasz,

Not a fix but more information about the issue.

patch_stdout.py is missing isatty as you observed, and should probably just pass through that property from the original stdout:

def isatty(self) -> bool: return self.original_stdout.isatty()

I believe the bytes-alike vs str issue is due to how the StdoutProxy class now works. The TypeErrors are the result of PT trying to get a cursor position report from the terminal by writing '\x1b[6n' to what it thinks is stdout .

output/vt100.py

def ask_for_cpr(self) -> None:
    """        Asks for a cursor position report (CPR).        """
    self.write_raw('\x1b[6n')
    self.flush()

So Vt100Output does a raw write to its internal buffer - fine so far - but when it calls flush() this happens:

output/vt100.py

        if self.write_binary:
            if hasattr(self.stdout, 'buffer'):
                out = self.stdout.buffer  # Py3.
            else:
                out = self.stdout
            out.write(data.encode(self.stdout.encoding or 'utf-8', 'replace'))
        else:
            self.stdout.write(data)

The first 'if' is True, the second False, so the buffer gets written to self.stdout which is a StdoutProxy object. The proxy then tries to split out newlines (to avoid excessive write() calls, see stdout_proxy.py:113), and expecting a 'str' gets a bytes-like instead, raising the TypeError. If the data variable at line 121 is examined it does hold the control code for cursor position.

StdoutProxy understands about writing binary vs not but only when invoked:

def __init__(self, raw: bool = False,
             original_stdout: Optional[TextIO] = None) -> None:
    ...
    self._raw = raw

However the _raw attribute is not actually used anywhere so it looks like the proxy needs to be extended to enable raw/cooked mode switching as needed. This is as far down the rabbit hole as I went. I don't know how Jonathan might want to fix this since 3.0 is in-progress so I haven't tried to fix it myself. I hope my analysis isn't too badly incorrect ;-)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/prompt-toolkit/python-prompt-toolkit/issues/934?email_source=notifications&email_token=AAB4N334OCHOD3MK7P2WEMDP5UJEBA5CNFSM4H5EUDQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZFWIQQ#issuecomment-508257346, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB4N36N3ELY3NVGXKD3CVTP5UJEBANCNFSM4H5EUDQA .

[sthibaul]

This is still an issue for me with 3.0.7.