os.system stops showing output after first use from worker thread (when running "python-wasm --worker")

[williamstein]

I just noticed this:

~/zython/packages/dash$ zython-dev   # or zython
Welcome to Node.js v18.10.0.
Type ".help" for more information.
> Python 3.11.0rc2 (main, Sep 30 2022, 11:52:38) [Clang 15.0.0 (git@github.com:ziglang/zig-bootstrap.git 9be8396b715b10f64d8a94b2 on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> import os         
>>> os.system("ls -l")
0
total 16
-rw-rw-r-- 1 wstein wstein 1947 Oct  1 08:27 Makefile
drwxrwxr-x 4 wstein wstein 4096 Oct  1 08:28 build
drwxrwxr-x 4 wstein wstein 4096 Oct  1 08:28 dist
drwxrwxr-x 3 wstein wstein 4096 Sep 28 23:58 src

>>> os.system("ls -l")  # no output
0

But with zython-debug it works fine. This suggests some subtle issue with fork_exec and using a worker thread. The likely best solution may be to completely eliminate using a worker thread by using getwchar for CLI use. I think that's just a matter of more effort to get that approach to work.

[williamstein]

This is still slightly broken, though mostly fixed after the extensive rewrite/refactor, and it is still a problem only when using a worker thread. Fortunately, that is now NOT the default (except on non-posix platforms where os.system isn't supported anyways). So I did implement "The likely best solution may be to completely eliminate using a worker thread by using getwchar for CLI use. " when it matters.

~/cowasm/packages/python-wasm$ ./bin/python-wasm 
Python 3.11.0 (main, Oct 27 2022, 10:03:11) [Clang 15.0.3 (git@github.com:ziglang/zig-bootstrap.git 0ce789d0f7a4d89fdc4d9571 on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.system("ls -l")
total 87920
-rw-r--r--    1 wstein  staff      3337 Oct 30 07:46 Makefile
-rw-r--r--    1 wstein  staff      1854 Oct 29 08:14 README.md
-rw-r--r--    1 wstein  staff        12 Oct 27 21:57 a.py
-rw-r--r--    1 wstein  staff  44287296 Oct 30 13:41 a.wat
drwxr-xr-x    3 wstein  staff        96 Oct 27 20:40 bin
drwxr-xr-x    3 wstein  staff        96 Oct 30 13:40 build
drwxr-xr-x   38 wstein  staff      1216 Oct 30 13:40 dist
-rw-r--r--    1 wstein  staff        76 Oct 27 20:40 index.d.ts
drwxr-xr-x  202 wstein  staff      6464 Oct 30 13:40 node_modules
-rw-r--r--    1 wstein  staff    698143 Oct 29 08:14 package-lock.json
-rw-r--r--    1 wstein  staff      1290 Oct 30 13:59 package.json
drwxr-xr-x   22 wstein  staff       704 Oct 30 13:40 src
-rw-r--r--    1 wstein  staff       208 Oct 27 20:40 tsconfig.json

0
>>> os.system("ls -l")
total 87920
-rw-r--r--    1 wstein  staff      3337 Oct 30 07:46 Makefile
-rw-r--r--    1 wstein  staff      1854 Oct 29 08:14 README.md
-rw-r--r--    1 wstein  staff        12 Oct 27 21:57 a.py
-rw-r--r--    1 wstein  staff  44287296 Oct 30 13:41 a.wat
drwxr-xr-x    3 wstein  staff        96 Oct 27 20:40 bin
drwxr-xr-x    3 wstein  staff        96 Oct 30 13:40 build
drwxr-xr-x   38 wstein  staff      1216 Oct 30 13:40 dist
-rw-r--r--    1 wstein  staff        76 Oct 27 20:40 index.d.ts
drwxr-xr-x  202 wstein  staff      6464 Oct 30 13:40 node_modules
-rw-r--r--    1 wstein  staff    698143 Oct 29 08:14 package-lock.json
-rw-r--r--    1 wstein  staff      1290 Oct 30 13:59 package.json
drwxr-xr-x   22 wstein  staff       704 Oct 30 13:40 src
-rw-r--r--    1 wstein  staff       208 Oct 27 20:40 tsconfig.json

0
>>> ^D
~/cowasm/packages/python-wasm$ ./bin/python-wasm --worker
Python 3.11.0 (main, Oct 27 2022, 10:03:11) [Clang 15.0.3 (git@github.com:ziglang/zig-bootstrap.git 0ce789d0f7a4d89fdc4d9571 on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> import os         
>>> os.system("ls -l")
0
total 87920
-rw-r--r--    1 wstein  staff      3337 Oct 30 07:46 Makefile
-rw-r--r--    1 wstein  staff      1854 Oct 29 08:14 README.md
-rw-r--r--    1 wstein  staff        12 Oct 27 21:57 a.py
-rw-r--r--    1 wstein  staff  44287296 Oct 30 13:41 a.wat
drwxr-xr-x    3 wstein  staff        96 Oct 27 20:40 bin
drwxr-xr-x    3 wstein  staff        96 Oct 30 13:40 build
drwxr-xr-x   38 wstein  staff      1216 Oct 30 13:40 dist
-rw-r--r--    1 wstein  staff        76 Oct 27 20:40 index.d.ts
drwxr-xr-x  202 wstein  staff      6464 Oct 30 13:40 node_modules
-rw-r--r--    1 wstein  staff    698143 Oct 29 08:14 package-lock.json
-rw-r--r--    1 wstein  staff      1290 Oct 30 13:59 package.json
drwxr-xr-x   22 wstein  staff       704 Oct 30 13:40 src
-rw-r--r--    1 wstein  staff       208 Oct 27 20:40 tsconfig.json

>>> os.system("ls -l")
0
>>> ^D

Note that subprocess.run works fine:

~/cowasm/packages/python-wasm$ ./bin/python-wasm --worker
Python 3.11.0 (main, Oct 27 2022, 10:03:11) [Clang 15.0.3 (git@github.com:ziglang/zig-bootstrap.git 0ce789d0f7a4d89fdc4d9571 on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess                                
>>> subprocess.run('ls', capture_output=True).stdout
b'Makefile\nREADME.md\na.py\nbin\nbuild\ndist\nindex.d.ts\nnode_modules\npackage-lock.json\npackage.json\nsrc\ntsconfig.json\n'
>>> subprocess.run('ls', capture_output=True).stdout
b'Makefile\nREADME.md\na.py\nbin\nbuild\ndist\nindex.d.ts\nnode_modules\npackage-lock.json\npackage.json\nsrc\ntsconfig.json\n'

So this is only a problem with os.system displaying output when running from a child thread, which sort of feels like the sort of thing that should not be supported anyways.