FastIO output gets truncated in windows terminal

[jaredliw]

Python Version: Python 3.7.7

Describe the bug Output gets truncated when printing a large amount of data over stdout (no problem with file I/O).

To Reproduce Copy FastIO.py from this repo to local, append these lines to the end of the file:

for x in range(5000):
    print(f"{x:0>7} Hello world!")

Expected behaviour It should print from 0000000 Hello world! to 0004999 Hello world!.

Additional context I change the lines above to this:

for x in range(5000):
    print(f"{x:0>7} Hello world!")
    if x % 1000 == 0:  # ofcoz, 1000 doesn't always work
        print(flush=True)

It works fine. This is error-prone. So I would like to ask whether it is possible to flush the output automatically whenever needed?

Thanks in advanced.

[bjorn-martinsson]

I'm not able to recreate your issue at all. The way fastio is intended to work (and works for me when I test it) is that the output should be flushed when python closes. Only thing that I can think of that would stop this is if you somehow crash python instead of exiting.

Can you tell me exactly what it is you are doing when running the script. I want your script, I want to know exactly what you wrote to run Python. If you are on windows or linux etc.

[jaredliw]

I ran it on Ubuntu VM, it worked fine. And I tried it once again on my other Windows PC, yeah, the same problem happened.

The code is as described above, nothing fancy. To dispel misunderstanding, I would just paste it here.

import os
import sys
from io import BytesIO, IOBase

_str = str
str = lambda x=b"": x if type(x) is bytes else _str(x).encode()

BUFSIZE = 8192

class FastIO(IOBase):
    newlines = 0

    def __init__(self, file):
        self._fd = file.fileno()
        self.buffer = BytesIO()
        self.writable = "x" in file.mode or "r" not in file.mode
        self.write = self.buffer.write if self.writable else None

    def read(self):
        while True:
            b = os.read(self._fd, max(os.fstat(self._fd).st_size, BUFSIZE))
            if not b:
                break
            ptr = self.buffer.tell()
            self.buffer.seek(0, 2), self.buffer.write(b), self.buffer.seek(ptr)
        self.newlines = 0
        return self.buffer.read()

    def readline(self):
        while self.newlines == 0:
            b = os.read(self._fd, max(os.fstat(self._fd).st_size, BUFSIZE))
            self.newlines = b.count(b"\n") + (not b)
            ptr = self.buffer.tell()
            self.buffer.seek(0, 2), self.buffer.write(b), self.buffer.seek(ptr)
        self.newlines -= 1
        return self.buffer.readline()

    def flush(self):
        if self.writable:
            os.write(self._fd, self.buffer.getvalue())
            self.buffer.truncate(0), self.buffer.seek(0)

class IOWrapper(IOBase):
    def __init__(self, file):
        self.buffer = FastIO(file)
        self.flush = self.buffer.flush
        self.writable = self.buffer.writable
        self.write = lambda s: self.buffer.write(s.encode("ascii"))
        self.read = lambda: self.buffer.read().decode("ascii")
        self.readline = lambda: self.buffer.readline().decode("ascii")

sys.stdin, sys.stdout = IOWrapper(sys.stdin), IOWrapper(sys.stdout)
input = lambda: sys.stdin.readline().rstrip("\r\n")

for x in range(5000):
    print(f"{x:0>7} Hello world!")

[bjorn-martinsson]

I use windows myself. I'm still not able to reproduce your issue.

What terminal are you using? Are you piping the output into a file? etc. There has to be some difference in our setup that causes this.

[bjorn-martinsson]

An update on this. I have been able to reproduce it now. I suspect that this is the culprit https://bugs.python.org/issue11395 (a known "bug" with windows console). Basically too big writes using os.write directly to the terminal gets truncated. The exact size when this happens seems to depend on a lot of parameters.

The workaround used inside Python is to check if you are using windows and if you are trying to write directly to a terminal, then limit the write to 32767 bytes. They also mention that this fix isn't perfect, and can break.

I technically could try to add some fall back for FastIO, that checks if you are on windows and write directly to a terminal, and in that case fall back to Pythons built in IO. But I'm not sure that would be worth it.

Since all of this is highly depent on the operating system and/or terminal. Could you please specify which version of windows you are using, and which terminal you are using. Also could you check whether redirecting the output into a pipe or a file causes truncation or not.

[jaredliw]

• Windows 10 Home Single Language Edition, version 20H2
• Terminal version 10.0.19042.1415
• Piping is OK

  python test.py > test.txt

• File I/O is OK

  file_output = open("test.txt", "w")
  sys.stdin, sys.stdout = IOWrapper(sys.stdin), IOWrapper(file_output)

[jaredliw]

If I write like this:

sys.stdin, sys.stdout = IOWrapper(sys.stdin), IOWrapper(open("test.txt", "w"))

it gives an error:

Exception ignored in: <__main__.IOWrapper object at 0x00000165229EBBC8>
Traceback (most recent call last):
  File "test.py", line 41, in flush
    write(self._fd, self.buffer.getvalue())
OSError: [Errno 9] Bad file descriptor

is it as expected?

[bjorn-martinsson]

If I write like this:

sys.stdin, sys.stdout = IOWrapper(sys.stdin), IOWrapper(open("test.txt", "w"))

it gives an error:

The reason why this fails is a stupid quirk with this specific implementation of FastIO. The problem is that since the file object is not stored anywhere, it gets garbage collected and is automatically closed. I told @cheran-senthil about this issue back when FastIO was added to pyrival, but he didn't bother fixing it/ didn't think it mattered. The fix is easy, just store the file object somewhere

    def __init__(self, file):
        self._file = file
        self._fd = file.fileno()