Open battaglia01 opened 1 year ago
ccordoba12
commented
1 year ago Hey @battaglia01, thanks for reporting. You said:
I have tested these in qtconsole and these bugs do not happen there
That's not true for me on Linux with your first example:
And I found the solution here after googling for jupyter problems os.fork:
I guess the solution should work similarly with your second example, so please try it and let us know.
battaglia01
commented
1 year ago Not here - still stuck. It doesn't get past the Popen call. Here it is in Spyder:
And it does work in qtconsole:
ccordoba12
commented
1 year ago Ok, this could be an issue that only happens on Mac. But I guess you're aware that you can also use the %run magic in Spyder to run your files (I mean, in the same way your screenshot shows that you're using it in Qtconsole). So, I'd suggest you do that if you want to work with os.fork.
@mrclary, are you seeing the same results with @battaglia01's first example on Mac?
mrclary
commented
1 year ago Okay, there are two issues and they are not related to Spyder.
time.sleep takes argument in seconds, not milliseconds: waiting for 2h46m may appear to be a hang but isn't.os.fork, it seems that it is not closed when execution is completed; use os._exit to close child processes.Regarding the first, I don't think you intended to wait 2 hours and 46 minutes. This could give the appearance of hanging when it is not. Changing to a more reasonable value fixes this issue. When processes are forked, it is arbitrary which lines of which processes are executed first, so print statements may not appear as expected.
Regarding the second, if the script is run from a non-interactive interpreter, e.g. $ python mwe_1.py, then the child process is destroyed when the parent Python process exits. However, when run from an interactive interpreter, e.g. an IPython console or Spyder's console or a Qtconsole, then the parent process does not exit and the child process is not destroyed closed. I observed this by viewing the Activity Monitor, but this can also be seen with $ ps -e | grep python. I don't know whether this is a bug or a feature of os.fork, but I observed this in both in IPython consoles started from the terminal and Qtconsoles. The solution is to use os._exit(0) to exit the child process at an appropriate point.
Interestingly, correcting the time.sleep in your first example, but not using os._exit(0), I encountered the following error regarding a bad file descriptor if using the magic %run command in an IPython console, but not in a Spyder console or Qtconsole:
In [1]: %run mwe_1.py
calling Popen
after Popen call
subprocess id: 57772
subprocess parent id: 57771
In [2]: Traceback (most recent call last):
File "/Users/rclary/opt/mambaforge/envs/spy-dev/bin/ipython", line 10, in <module>
sys.exit(start_ipython())
File "/Users/rclary/opt/mambaforge/envs/spy-dev/lib/python3.10/site-packages/IPython/__init__.py", line 124, in start_ipython
return launch_new_instance(argv=argv, **kwargs)
File "/Users/rclary/opt/mambaforge/envs/spy-dev/lib/python3.10/site-packages/traitlets/config/application.py", line 1043, in launch_instance
app.start()
File "/Users/rclary/opt/mambaforge/envs/spy-dev/lib/python3.10/site-packages/IPython/terminal/ipapp.py", line 318, in start
self.shell.mainloop()
File "/Users/rclary/opt/mambaforge/envs/spy-dev/lib/python3.10/site-packages/IPython/terminal/interactiveshell.py", line 887, in mainloop
self.interact()
File "/Users/rclary/opt/mambaforge/envs/spy-dev/lib/python3.10/site-packages/IPython/terminal/interactiveshell.py", line 872, in interact
code = self.prompt_for_code()
File "/Users/rclary/opt/mambaforge/envs/spy-dev/lib/python3.10/site-packages/IPython/terminal/interactiveshell.py", line 811, in prompt_for_code
text = self.pt_app.prompt(
File "/Users/rclary/opt/mambaforge/envs/spy-dev/lib/python3.10/site-packages/prompt_toolkit/shortcuts/prompt.py", line 1035, in prompt
return self.app.run(
File "/Users/rclary/opt/mambaforge/envs/spy-dev/lib/python3.10/site-packages/prompt_toolkit/application/application.py", line 961, in run
return loop.run_until_complete(coro)
File "/Users/rclary/opt/mambaforge/envs/spy-dev/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete
self.run_forever()
File "/Users/rclary/opt/mambaforge/envs/spy-dev/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/Users/rclary/opt/mambaforge/envs/spy-dev/lib/python3.10/asyncio/base_events.py", line 1868, in _run_once
event_list = self._selector.select(timeout)
File "/Users/rclary/opt/mambaforge/envs/spy-dev/lib/python3.10/selectors.py", line 562, in select
kev_list = self._selector.control(None, max_ev, timeout)
OSError: [Errno 9] Bad file descriptor
If you suspect this is an IPython 8.11.0 bug, please report it at:
https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@python.org
You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.
Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
%config Application.verbose_crash=True
In [2]: This same error occurs if you execute os.fork() on a single line in an interactive interpreter.
So, the following is what I recommend for your first example script, which works using both F5 in Spyder, copy-and-paste into the console, as well as %run mwe_1_fixed.py.
# mwe_1_fixed.py
import os
import subprocess as sp
import time
pid = os.fork()
if pid == 0:
print("calling Popen")
p = sp.Popen(["python", "testchild.py"]) # Does not hang here
print("after Popen call")
time.sleep(1) # 1 second, not 2h46m
os._exit(0) # Be sure to exit child if parent is interactive interpreter
else:
time.sleep(1) # 1 second, not 2h46mHowever, I prefer the following examples scripts that identify everything.
# mwe_1_works.py
import os
import subprocess as sp
pid = os.fork()
if pid == 0:
try:
print("I'm child with ID:", os.getpid())
print("My parent's ID:", os.getppid())
p = sp.Popen(["python", "testchild.py"], stdout=sp.PIPE, stderr=sp.PIPE) # Does not hang
print(p.communicate()[0].decode()) # Print the subprocess output
finally:
# Be sure to exit child process if running in interactive interpreter,
# even when there is an error.
os._exit(0)
else:
print("I'm parent with ID:", os.getpid())
print("My child's ID:", pid)# testchild.py
import os
print('subprocess id:', os.getpid())
print('subprocess parent id:', os.getppid())And similarly for your second example:
# mwe_2_works.py
import os
import time
pid1 = os.fork()
if pid1 == 0:
print("child 1:", os.getpid())
print("parent of ch1:", os.getppid())
pid2 = os.fork() # Does not hang
print("child 1 post-second-fork") # prints twice: pid1 and pid2
if pid2 == 0:
try:
print("child 2:", os.getpid())
os.execlp("python", "python", "testchild.py") # replaces child 2 process
finally:
# The subprocess closes fine; since it replaced the child 2 process,
# there is no need to close child 2 here, but is still good practice
os._exit(0)
else:
time.sleep(1)
os._exit(0) # close child 1
else:
time.sleep(2)
print("parent:", os.getpid())Yes, I'm aware that 10000 seconds is a long time. I did that on purpose to keep the parent process alive.
No, that isn't the cause of the problem. For starters, the code snippet you provided also does not work:
# mwe_1_fixed.py
import os
import subprocess as sp
import time
pid = os.fork()
if pid == 0:
print("calling Popen")
p = sp.Popen(["python", "testchild.py"]) # Does not hang here
print("after Popen call")
time.sleep(1) # 1 second, not 2h46m
os._exit(0) # Be sure to exit child if parent is interactive interpreter
else:
time.sleep(1) # 1 second, not 2h46mThe child process hangs on the line which you have commented "Does not hang here." The parent process then terminates after one second. The child process remains open and is hung on the Popen call. testchild.py does not execute.
You've written a very long post to respond to but it doesn't seem the premise of it is correct; there are other statements you make which are also not true on my system. I will again reiterate that, on M1, this only happens within Spyder and not in qtconsole, IPython, CPython or anything else.
Before we continue, have you tried to actually run this on an M1 Mac?
mrclary
commented
1 year ago @battaglia01, I performed all my tests with macOS 13.2.1 with an M1 chip and macOS 12.6.3 with Intel chip; both with identical results.
My terminal environment uses Python 3.10.9 and IPython 8.11.0. So I've tested a few different versions.
The obvious next question is what is the content of your testchild.py?
there are other statements you make which are also not true on my system.
Please elaborate.
battaglia01
commented
1 year ago Please elaborate.
For starters, we do not seem to be on the same basic page about how fork() and sleep() work.
There are two sleep commands in my original post. The first is in the child process and after the line which hangs (and does not seem to ever be executed at all). The second is in the parent process and has nothing to do with code execution in the child process. I am not sure which you are saying is causing Popen to hang.
Regarding the one in the parent, there is no possible mechanism whereby calling sleep() in one process would block in an entirely different, forked child process, unless the spyder kernel is doing something very strange. That would be a much worse bug than the one I am reporting.
The one in the child is also irrelevant; you can replace subprocess.Popen() with subprocess.run(), which is a blocking call. It does not run testchild.py at all and simply hangs there. The fact that it hangs also on Popen is much more interesting, since Popen is not a blocking call.
# mwe_1_works.py
import os
import subprocess as sp
pid = os.fork()
if pid == 0:
try:
print("I'm child with ID:", os.getpid())
print("My parent's ID:", os.getppid())
p = sp.run(["python", "testchild.py"]) # Hangs on my system
finally:
# Be sure to exit child process if running in interactive interpreter,
# even when there is an error.
os._exit(0)
else:
print("I'm parent with ID:", os.getpid())
print("My child's ID:", pid)testchild.py is never called on my system at all. I am not sure why it runs on yours. I'll try on a different one later and see what happens.
battaglia01
commented
1 year ago FWIW, my testchild.py is just a simple script:
testchild.py just prints something every second and writes to a logfile:
import sys, time, os
from datetime import datetime
def write_both(s, end="\n", flush=True, prefix=None):
print(s, end=end, flush=flush)
if prefix is None:
prefix = str(datetime.now())
with open("/tmp/testlog", "a+") as f:
f.write(prefix + " " + s + end)
write_both("From child process: I have PID = %d" % os.getpid())
while True:
try:
write_both("Printing from child process! PID = %d" % os.getpid())
time.sleep(1)
except KeyboardInterrupt:
write_both("Ctrl-C pressed in child process! PID = %d" % os.getpid())
write_both("IT WORKED!")
sys.exit(0)But again, as we do not seem to agree on the basics of how fork and sleep work, I'd rather focus on the above first.
mrclary
commented
1 year ago @battaglia01, I agree that the parent process sleep should not affect the child process. I was referring to the one in the child process. I was not aware of what was in your testchild.py, so it was not clear exactly how you distinguished between hanging and the egregious sleep time. So we do not have any disagreement on how sleep and fork work.
Nevertheless, I see your testchild.py and I'll test it.
Are you using the "Same as Spyder" interpreter or are you using a different external environment?
mrclary
commented
1 year ago Okay, I see that there is a problem on M1. I still don't know exactly what the problem is, but I can reproduce the issue only on M1. The interesting thing is that the MWE works just fine on M1 for the Spyder environment installed by our experimental installer, but not for any other environment. This is also independent of the Spyder launch mechanism. I will investigate further. It will either be a package version problem or a conda M1 build problem; hopefully the former.
ccordoba12
commented
1 year ago The interesting thing is that the MWE works just fine on M1 for the Spyder environment installed by our experimental installer, but not for any other environment.
Ok, this is very important and it seems @battaglia01 omitted to mention it before. Its significance is that we use a shell script to activate other external environments, which could be causing issues with os.fork (although I still can't reproduce the problem on Linux, even for external envs).
battaglia01
commented
1 year ago I have Spyder installed in its own conda environment; I didn't use the Mac installer from the website. I get the problem with any interpreter, including the one in the spyder-env environment. I also get the problem if preferences are reset. I haven't tried the version from the Mac installer.
On Sun, Mar 5, 2023 at 2:33 PM Carlos Cordoba @.***> wrote:
The interesting thing is that the MWE works just fine on M1 for the Spyder environment installed by our experimental installer, but not for any other environment.
Ok, this is very important and it seems @battaglia01 https://github.com/battaglia01 omitted to mention it before. Its significance is that we use a shell script to activate other external environments, which could be causing issues with os.fork (although I still can't reproduce the problem on Linux, even for external envs).
— Reply to this email directly, view it on GitHub https://github.com/spyder-ide/spyder/issues/20629#issuecomment-1455182475, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHZFMWF3ZO2URRHKPQJE2TW2TTBZANCNFSM6AAAAAAVPCO3IE . You are receiving this because you were mentioned.Message ID: @.***>
mrclary
commented
1 year ago @ccordoba12, I want to be sure to clarify.
While I still need to investigate further, the above suggests a package version issue and/or conda M1 package build issue. But I will also investigate the shell script for launching the external environments.
battaglia01
commented
1 year ago What "experimental" environment are you referring to?
On Sun, Mar 5, 2023 at 3:56 PM Ryan Clary @.***> wrote:
@ccordoba12 https://github.com/ccordoba12, I want to be sure to clarify.
- I cannot reproduce this issue in any way on macOS 12.6.3 with Intel. I think it only occurs on M1 chip machines.
- Any launch mechanism for Spyder (standalone, conda, experimental) will work on M1 if they use the experimental conda-based environment for the interpreter. Any other environment exhibits the OP behavior.
While I still need to investigate further, the above suggests a package version issue and/or conda M1 package build issue. But I will also investigate the shell script for launching the external environments.
— Reply to this email directly, view it on GitHub https://github.com/spyder-ide/spyder/issues/20629#issuecomment-1455203067, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHZFMVFV6FZX6A4QGQSMHTW2T4WBANCNFSM6AAAAAAVPCO3IE . You are receiving this because you were mentioned.Message ID: @.***>
mrclary
commented
1 year ago What "experimental" environment are you referring to?
We have an experimental installer (https://github.com/spyder-ide/spyder/releases/download/v5.4.2/EXPERIMENTAL-Spyder-5.4.2-macOS-x86_64.pkg) which installs an independent conda environment from which to run Spyder and creates a Spyder.app shortcut to launch Spyder. This is the environment to which I'm referring.
mrclary
commented
1 year ago Okay, I've got some new information. Creating any environment with x86_64 architecture will result in correct behavior on M1, while creating any environment with arm64 architecture will manifest the OP.
You can test this by creating an environment as follows:
$ CONDA_SUBDIR=osx-64 conda create -n test-x86 spyder-kernelsUsing this environment for your Python interpreter in Spyder, regardless of the nature of your Spyder installation will result in correct behavior.
However, as @battaglia01 has reported, the OP only manifests in Spyder. An arm64 environment will behave correctly when running the script in IPython or Qtconsole interpreters via %run.
@ccordoba12, I have no idea why this would be the case. Any ideas?
ccordoba12
commented
1 year ago Creating any environment with x86_64 architecture will result in correct behavior on M1, while creating any environment with arm64 architecture will manifest the OP.
Thanks for digging deep into this issue @mrclary! I had no idea it was so architecture dependent.
I have no idea why this would be the case. Any ideas?
Nop, I'm also clueless about this. Perhaps @impact27 has more insights about this.
mrclary
commented
1 year ago @impact27, since spyder-kernels is noarch, I think this issue may not be a direct result of spyder-kernels itself. Furthermore, since %run for an arm64 environment is successful in IPython and Qtconsole from a Terminal, I don't think this issue is a direct result of ipython or qtconsole builds, either. So I suspect that there may be something about the way spyder-kernels interacts with qtconsole; perhaps a spyder-kernels dependency has a faulty M1 build?
impact27
commented
1 year ago Running:
import os
import subprocess as sp
pid = os.fork()
if pid == 0:
try:
print("child")
p = sp.run(["echo", "subprocess"]) # Hangs on my system
finally:
# Be sure to exit child process if running in interactive interpreter,
# even when there is an error.
os._exit(0)
else:
print("parent")I get a crash that informs me that:
Exception Type: EXC_BREAKPOINT (SIGTRAP)
Exception Codes: 0x0000000000000001, 0x00000001af96a0e0
Termination Reason: Namespace SIGNAL, Code 5 Trace/BPT trap: 5
Terminating Process: exc handler [68684]
Application Specific Information:
*** multi-threaded process forked ***
BUG IN CLIENT OF LIBPLATFORM: os_unfair_lock is corrupt
Abort Cause 258
crashed on child side of fork pre-exec
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_platform.dylib 0x1af96a0e0 _os_unfair_lock_corruption_abort + 88
1 libsystem_platform.dylib 0x1af965024 _os_unfair_lock_lock_slow + 328
2 libtcl8.6.dylib 0x13213df24 AtForkPrepare + 44
3 libsystem_pthread.dylib 0x1af938744 _pthread_atfork_prepare_handlers + 96
4 libSystem.B.dylib 0x1bafd151c libSystem_atfork_prepare + 32
5 libsystem_c.dylib 0x1af81fa28 fork + 36
6 _posixsubprocess.cpython-310-darwin.so 0x1016cae28 do_fork_exec + 68
7 _posixsubprocess.cpython-310-darwin.so 0x1016ca944 subprocess_fork_exec + 796
8 Python 0x101206498 cfunction_call + 96I can reproduce in python:
save in a file test.py:
import turtle
import os
import subprocess as sp
pid = os.fork()
if pid == 0:
try:
print("child")
p = sp.run(["pwd"]) # Hangs on my system
finally:
# Be sure to exit child process if running in interactive interpreter,
# even when there is an error.
os._exit(0)
else:
print("parent")run with python3 test.py
so turtle is the culprit ??????????
impact27
commented
1 year ago This is a python bug (https://github.com/python/cpython/issues/102525), so I am unsure what we should do. @ccordoba12 should we remove turtle from spydercustomize?
battaglia01
commented
1 year ago Same here! Good figuring this out. I ran a slightly modified version
import turtle
import os
import subprocess as sp
pid = os.fork()
if pid == 0:
try:
print("child")
p = sp.Popen(["pwd"]) # Hangs on my system
print("post pwd")
finally:
# Be sure to exit child process if running in interactive interpreter,
# even when there is an error.
os._exit(0)
else:
print("parent")Same thing; this crashes CPython. So yeah, looks like importing turtle somehow does this...?
mrclary
commented
1 year ago @battaglia01 @impact27, I corroborate your latest observations. Additionally, I find that Spyder's IPython console reports the wrong processor:
In [1]: import platform
In [2]: platform.processor()
Out [2]: 'i386'Where a Qtconsole initiated from the same environment reports 'arm'.
mrclary
commented
1 year ago @impact27 @ccordoba12, I can also confirm that in the arm environment, If I comment out the turtle section in spyder_kernels.customize.spydercustomize.py, the OP is remediated. However, even if turtle is removed, I assume that this issue would reemerge if users ever import tkinter in the environment (see https://github.com/python/cpython/issues/102525#issuecomment-1460848609)
mrclary
commented
1 year ago Also, there would be a regression of #6278 and #11967. However, I find issues with running turtle even in an IPython console independent of Spyder. IMHO, turtle is just not designed to be run in an interactive console, let alone multiple times in the same interactive console, so I don't think it should be the responsibility of Spyder developers to monkey-patch turtle.
ccordoba12
commented
1 year ago Also, there would be a regression of https://github.com/spyder-ide/spyder/issues/6278 and https://github.com/spyder-ide/spyder/issues/11967.
True.
However, I find issues with running turtle even in an IPython console independent of Spyder.
What issues? (Please open a new issue about it to not derail the discussion here).
IMHO, turtle is just not designed to be run in an interactive console
Well, we had to patch turtle to make it work in Spyder because many people complained about it. So, the solution is to keep that patch, fix the issues you found, and provide a way to disable it using an option in the Preferences section of the IPython console.
Issue Report Checklist
conda update spyder(orpip, if not using Anaconda)jupyter qtconsole(if console-related)spyder --resetProblem Description
There are several problems using os.fork() in Spyder if one tries to spawn any additional process from the forked process. This is on macOS M1; I am sure things may vary on other platforms.
I have tested these in qtconsole and these bugs do not happen there. I thought some of this may be related to qtconsole being a tty emulator, but apparently not. It's something related to Spyder.
Things are broken in a lot of very subtle ways and I am not quite sure how to list them all. Here are a few:
It hangs on Popen; the line after never prints. Another one:
Again, the second os.fork() command completely hangs.
There are quite a few similar situations above, most of which involves running another subprocess in some form from the forked process.
There are about a dozen other quirks like this, including one particularly subtle one involving a pty which threw me off for hours earlier this week. I am not sure to what extent these are "different" bugs or manifestations of the same bug. Is there some big picture to this?
Versions
Dependencies