cmd-action fails with BlockingIOError: [Errno 11] write could not complete without blocking

[jjarmagost]

Describe the bug Command action uses verbosity 2, and invokes an executable that produces a lot of output in the terminal. (The executable is part of an FPGA simulation flow, but I believe the actual executable is irrelevant.) After making it most of the way through, the output abruptly stops with the following:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/jjarmago/.local/lib/python3.6/site-packages/doit/action.py", line 179, in _print_process_output
    realtime.write(line)
BlockingIOError: [Errno 11] write could not complete without blocking
Unhandled exception in thread started by <bound method Thread._bootstrap of <Thread(Thread-1, started 139762011436800)>>
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/jjarmago/.local/lib/python3.6/site-packages/doit/action.py", line 179, in _print_process_output
    realtime.write(line)
BlockingIOError: [Errno 11] write could not complete without blocking

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 884, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.6/threading.py", line 926, in _bootstrap_inner
    (self.name, _format_exc()), file=_sys.stderr)
BlockingIOError: [Errno 11] write could not complete without blocking

There are records on the internet of other people having similar problems with a variety of different tools:

https://stackoverflow.com/questions/54185874/logging-chokes-on-blockingioerror-write-could-not-complete-without-blocking

https://github.com/travis-ci/travis-ci/issues/8982

https://bugs.launchpad.net/plainbox/+bug/1401886

One of the records above suggests fixing the problem by reducing the amount of output printed to the terminal, but this is obviously just a hack. And furthermore, in my case the executable has no option to reduce verbosity.

Through experimentation I have discovered a workaround of using a small delay like this:

    def _print_process_output(self, process, input_, capture, realtime):
        """Reads 'input_' until process is terminated.
        Writes 'input_' content to 'capture' (string)
        and 'realtime' stream
        """
        if self.buffering:
            read = lambda: input_.read(self.buffering)
        else:
            # line buffered
            read = lambda: input_.readline()
        while True:
            try:
                line = read().decode(self.encoding, self.decode_error)
            except:
                # happens when fails to decoded input
                process.terminate()
                input_.read()
                raise
            if not line:
                break
            capture.write(line)
            if realtime:

                time.sleep(0.01)

                realtime.write(line)
                realtime.flush() # required if on byte buffering mode

But I don't know how this fixes the problem because I'm not an expert on Python threads or streams.

Please include a minimal dodo.py that reproduces the problem. If relevant also include the command line used to invoke doit.

Obviously without the actual FPGA simulation tool the real dodo.py is pretty worthless, but I'll see if I can reproduce the problem with pure Python or some commonly available executable.

Environment

1. OS:

NAME="Linux Mint" VERSION="19 (Tara)" ID=linuxmint ID_LIKE=ubuntu PRETTY_NAME="Linux Mint 19" VERSION_ID="19" HOME_URL="https://www.linuxmint.com/" SUPPORT_URL="https://forums.ubuntu.com/" BUG_REPORT_URL="http://linuxmint-troubleshooting-guide.readthedocs.io/en/latest/" PRIVACY_POLICY_URL="https://www.linuxmint.com/" VERSION_CODENAME=tara UBUNTU_CODENAME=bionic

2. python version:

Python 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0] on linux

3. doit version:

0.32.0 -- This is what pip installs