Output of os.system, subprocess.call goes to command prompt instead of qtconsole, on Windows 10, Open Suse

[kbauer]

Problem Description

When running os.system or subprocess.call on Windows 10, no output is produced in qtconsole. Instead the output goes to the Anaconda Prompt console or, if qtconsole was started from the start menu, is simply discarded.

The same issue is reproducible with ipython qtconsole and and the sypder IDE, where this issue was originally reported (https://github.com/spyder-ide/spyder/issues/14783), but declared an depenency issue.

When running bare-bones consoles (ipython, python), the issue does not appear, presumably because everything is displayed in the same cmd.exe window.

On OpenSuse the issue occurs likewise. If the console is started from a terminal window, that terminal will receive the output of the commands. When originally posted, I thought I couldn't reproduce the issue on Linux.

Qt Console

As a component in Spyder

What steps reproduce the problem?

1. Run jupyter qtconsole from the Anaconda prompt.
2. In the qtconsole type import os; os.system(r"mkdir C:\tmp\demo & dir C:\tmp\demo")

What is the expected output? What do you see instead?

The expectation is to see the output in the graphical console, together with the output of print or sys.stdout.write.

Instead the output is displayed in the command prompt window.

Potentially relevant package versions

• Python version: 3.8.5
• Qt version: 5.9.7
• PyQt version: 5.9.2, PyQt5: 5.12.3
• QtConsole: 5.0.2
• OS: Windows 10, Version 2004 (OS Build 19041.804)

DEPENDENCY REPORT OF SPYDER
atomicwrites >=1.2.0            :  1.4.0 (OK)
chardet >=2.0.0                 :  4.0.0 (OK)
cloudpickle >=0.5.0             :  1.6.0 (OK)
diff_match_patch >=20181111     :  20200713 (OK)
intervaltree >=3.0.2            :  3.1.0 (OK)
IPython >=7.6.0                 :  7.20.0 (OK)
jedi =0.17.2                    :  0.17.2 (OK)
jsonschema >=3.2.0              :  3.2.0 (OK)
keyring >=17.0.0                :  22.0.1 (OK)
nbconvert >=4.0                 :  6.0.7 (OK)
numpydoc >=0.6.0                :  1.1.0 (OK)
paramiko >=2.4.0                :  2.7.2 (OK)
parso =0.7.0                    :  0.7.0 (OK)
pexpect >=4.4.0                 :  4.8.0 (OK)
pickleshare >=0.4               :  0.7.5 (OK)
psutil >=5.3                    :  5.8.0 (OK)
pygments >=2.0                  :  2.7.4 (OK)
pylint >=1.0                    :  2.6.0 (OK)
pyls >=0.36.2;<1.0.0            :  0.36.2 (OK)
pyls_black >=0.4.6              :  0.4.6 (OK)
pyls_spyder >=0.3.0             :  0.3.0 (OK)
qdarkstyle >=2.8                :  2.8.1 (OK)
qtawesome >=0.5.7               :  1.0.1 (OK)
qtconsole >=5.0.1               :  5.0.2 (OK)
qtpy >=1.5.0                    :  1.9.0 (OK)
rtree >=0.8.3                   :  0.9.4 (OK)
setuptools >=39.0.0             :  52.0.0.post20210125 (OK)
sphinx >=0.6.6                  :  3.4.3 (OK)
spyder_kernels >=1.10.1;<1.11.0 :  1.10.1 (OK)
textdistance >=4.2.0            :  4.2.1 (OK)
three_merge >=0.1.1             :  0.1.1 (OK)
watchdog >=0.10.3               :  1.0.2 (OK)
zmq >=17                        :  20.0.0 (OK)

cython >=0.21                   :  0.29.21 (OK)
matplotlib >=2.0.0              :  3.3.4 (OK)
numpy >=1.7                     :  1.19.2 (OK)
pandas >=1.1.1                  :  1.2.1 (OK)
scipy >=0.17.0                  :  1.6.0 (OK)
sympy >=0.7.3                   :  1.7.1 (OK)

[kbauer]

I can observe the issue on Linux too after all, with one difference: When executing the commands in Spyder's console pane, the output doesn't appear in the terminal from which Spyder was started either.

[ccordoba12]

This is certainly a problem on Windows and I think it's not possible to fix it in the Jupyter architecture. I mean, there's no workaround or fix I'm aware of.

On Linux I can't reproduce it while using Spyder:

But that's because the kernels in our IPython console depend on the wurtlizer package, which takes care of redirecting stdout/stderr to the console.

Quick question (just to check): Are you using the Windows Subsystem for Linux or a real Linux installation/VM?

[kbauer]

Quick question (just to check): Are you using the Windows Subsystem for Linux or a real Linux installation/VM?

The Linux environment is a separate desktop at work, running Open Suse with Anaconda Python and spyder/jupyter updated with pip3 install --upgrade, then starting Spyder/Jupyter QtConsole from the terminal in order to ensure that it isn't the more dated zypper version I am starting.

When I first posted this ticket though, I didn't see the issue on OpenSuse, only on Windows. I am not sure if any relevant packages have changed in between. There is a difference between the native Windows installation and the native OpenSuse installation, but I don't have a snapshot of the versions I had on OpenSuse, when the issue did not occur. I would assume they were the same, but I don't know if pip3 upgrades dependencies, that are already installed with a sufficiently but not up-to-date version.

[kbauer]

It looks like wurlitzer isn't working the same in qtconsole as in terminals. (OpenSuse)

On Windows 10, it doesn't seem to be used at all and when installing it manually will fail, complaining about a missing package fnctl.

Output

(in ipython3 on terminal)        (on terminal directly)           (in qtconsole)
In [1]: %run /tmp/a.py           >>> python3 /tmp/a.py            In [3]: %run /tmp/a.py      
____ Uncaptured output ____      ____ Uncaptured output ____      ____ Uncaptured output ____ 
____ Captured Stdout ____        ____ Captured Stdout ____        print function, stdout      
print function, stdout           print function, stdout           print function, stderr      
print function, stderr           print function, stderr           ____ Captured Stdout ____   
os.system, stdout                os.system, stdout                os.system, stdout           
subprocess.call, stdout          subprocess.call, stdout          subprocess.call, stdout     

____ Captured Stderr ____        ____ Captured Stderr ____        ____ Captured Stderr ____   
os.system, stderr                os.system, stderr                os.system, stderr           
subprocess.call, stderr          subprocess.call, stderr          subprocess.call, stderr     

____ end ____                    ____ end ____                    ____ end ____               

The script /tmp/a.py

import os, wurlitzer, io, subprocess

out = io.StringIO()
err = io.StringIO()

print("____ Uncaptured output ____")
with wurlitzer.pipes(stdout=out, stderr=err):
    print("print function, stdout")
    print("print function, stderr")
    os.system("echo os.system, stdout")
    os.system("echo os.system, stderr >&2")
    subprocess.call(["bash", "-c", "echo subprocess.call, stdout"])
    subprocess.call(["bash", "-c", "echo subprocess.call, stderr >&2"])

print("____ Captured Stdout ____")
print(out.getvalue())
print("____ Captured Stderr ____")
print(err.getvalue())
print("____ end ____")

Version notes

>>> pip3 list | grep wurl
wurlitzer                          2.0.1

[ccordoba12]

Wurlitzer only works on Posix systems (sorry if I wasn't clear enough about it).

[impact27]

PR spyder-ide/spyder#1478 solves that problem, but only for consoles started from spyder. Also, if another client connects to the same console, it wouldn't recieve the messages either.

The ipython kernel creates a new sys.stdout. Any output that is written to sys.stdout or sys.stderr, for example using the print function, is sent to the client. The problem is that any non-python code is not aware of the new sys.stdout and uses the regular cout. On UNIX you can pipe that back to sys.stdout (wurlitzer does that) but I don't know of any way to do that on windows. In spyder we can control the process for the kernel (if it is started from spyder) and therefore the cout can be printed in the console (spyder-ide/spyder#14783) but this is not going through the regular ipython mechanisms. That means that the cout is not sent to any other clients.