stdout and stderr are non-blocking

[glyph]

If you put this code very close to the beginning of your python code in py2app:

import fcntl, os
print(fcntl.fcntl(1, fcntl.F_GETFL) & os.O_NONBLOCK, fcntl.fcntl(2, fcntl.F_GETFL) & os.O_NONBLOCK))

and run it in a terminal, you will see 4 4, indicating that the stdout and stderr streams have been set to non-blocking mode for some reason. This means (among other things) tracebacks will be irritatingly truncated when debugging in this way. It's easy enough to fix—just for fd in [1, 2]: fcntl.fcntl(fd, fcntl.F_SETFL, fcntl.fcntl(fd, fcntl.F_GETFL) | os.O_NONBLOCK)—but this is annoying out-of-the-box behavior and very obscure.

Is this something that AppKit or LaunchServices are doing on one's behalf, somehow? I don't know how they'd get involved before NSApplicationMain when you're just running the app as a binary in the terminal; is it behavior in the py2app stub?

[ronaldoussoren]

It could be behaviour in the py2app stub, but I don't explicitly set nonblocking mode there.

I don't get this behaviour with the project in examples/tkinter/hello_tk in the py2app source tree (macOS 12, python 3.10). That's using py2app from the repository, but that has no changes to the stub executable.

The option redirect_stdout_to_asl does affect stdout and stderr, but that option is off by default and redirects both streams to Apple's logging system (and has the side effect of no longer using the stdout stream).

[glyph]

I'm using Twisted in this app and it's possible that something in there is doing this — although it shouldn't be, that's only supposed to happen with StandardIO (and I'm not using that).

I'll try to create a minimal reproducer.

[glyph]

OK, this gets even weirder. I definitely narrowed it down to a minimal reproducer and it was definitely not what I thought. The bug is in py2app (or maybe distutils?) but not in anything runtime-related.

If you make a shell script that runs python setup.py py2app, that sets stdout to be non-blocking. I set up a repo here https://github.com/glyph/py2app-nonblocking-example for a minimal reproducer. If you just build the app and run it in your terminal, no problem (i.e. "0 0") but if you run the included shell script, it's broken ("4 4"). Unless you modify python setup.py py2app to be python setup.py py2app | cat which interposes something other than a tty onto stdout and makes the non-blocking-ness not stick around.

This doesn't happen with, e.g. python setup.py egg_info so it's not trivially distutils's fault.

I think my shell (zsh) is resetting the blocking state of the terminal on every interactive prompt, I think, which is why this doesn't happen interactively.

[glyph]

Weirder still. This affects builds themselves. The output from --alias is short enough that I don't get this, but a regular python setup.py py2app on https://github.com/glyph/Pomodouroboros results in this crash:

BlockingIOError: [Errno 35] write could not complete without blocking

but python setup.py py2app | cat works properly.

[ronaldoussoren]

This is very weird indeed, but finding the root cause might help fix other build issues. Py2app contains some retry loops for subcommands that fail, and IIRC at least some of them were added due to similar errors.

Are you on an M1 based system, or at least use a Python build with arm64 support (such as the 3.10 installers on Python.org)?

[ronaldoussoren]

Turns out this is a mis-feature of the ibtools(1) command used to compile NIB/XIB files. The changeset above fixes this and removes an older workaround for broken builds.

I'm not closing this issue yet because I'm thinking about creating a branch with a hot fix in the 0.28 release. I'm not comfortable yet about releasing from the tip of the tree, I'm partway through code cleanup and am not 100% sure that a release would be problem free.

And finally: Thanks for the reproducer, that made it a lot easier to debug the issue.

[ronaldoussoren]

Sigh.

For some reason my fixed worked for some time, but working on a back port somehow broke things again.

To be continued...

[ronaldoussoren]

323fe55a272e42e3d7beb85e07bac1afd7a7cbeb, 43422efda1c1bedcd258657d876f9bc21ec479e9, d9b37ed2085fbfb80cb0963680d19be736bee4f8

The previous "sigh" was of my own doing, the code that resets blocking state was buggy.

Both master and v0.28-branch seem to work correctly now. Current plan is to push a release from the v0.28-branch later today.

[ronaldoussoren]

This should be fixed in py2app 0.28.2 which is on PyPI.

[glyph]

@ronaldoussoren thank you for fixing this! I independently discovered the ibtool weirdness and was coming back here excited to report my findings :).