search

jdtsmith / python-mls

Multi-line Shell Commands for Python REPLs in Emacs
GNU Affero General Public License v3.0
40 stars 3 forks source link

Emacs hangs when entering a line that expects an indented block. #8

Closed joostkremers closed 2 years ago

joostkremers commented 2 years ago

When I type an expression into an inferior Python buffer that expects an indented code block, Emacs hangs the moment I hit RET. C-g or ESC ESC ESC doesn't get me out of it, I need to kill the process through the task manager.

This is in Emacs 27.2 on Windows. On Linux, with mostly the same setup, everything works.

Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> for i in range(10):

At this point, the moment I type RET, point jumps to the next line and then Emacs hangs. The continuation prompt (...) does not appear.

jdtsmith commented 2 years ago

I suspect this comes from the attempt to cancel the current command. Try M-: (python-mls-interrupt-quietly) and see if your process hangs. Please also report the value of python-mls-continuation-prompt in your shell buffer.

joostkremers commented 2 years ago

I suspect this comes from the attempt to cancel the current command. Try M-: (python-mls-interrupt-quietly) and see if your process hangs.

It does, but I am able to C-g out of it, which isn't possible in the inferior Python shell.

Please also report the value of python-mls-continuation-prompt in your shell buffer.

Doing C-h v python-mls-continuation-prompt when inside the inferior Python buffer gives me this:

python-mls-continuation-prompt is a variable defined in ‘python-mls.el’.
Its value is #("... " 0 4 (font-lock-face comint-highlight-prompt))

Documentation:
Current computed continuation prompt.
jdtsmith commented 2 years ago

Ok that’s useful. Maybe the windows process isn’t responding to the interrupt. Try just python-mls-interrupt, perhaps after a long sleep command. If that doesn’t work, try just C-c C-c In the same situation.

joostkremers commented 2 years ago

Actually, once Emacs hangs, there's nothing I can do anymore... C-g doesn't do anything, nor does C-c C-c, even after waiting a bit. (IME, comint buffers sometimes need a little time to process C-c C-c, but it this case, it never happens.) Emacs is completely stuck, I can't even click the close button in the window bar. I need to open the task manager to kill the process. (The process actually runs at about 30% CPU and makes the fans of my laptop spin up...)

I tried setting a timer in an IELM buffer with (run-with-timer 10 nil #'python-mls-interrupt), but even if that's the correct way of doing it, it didn't have any effect either...

jdtsmith commented 2 years ago

OK, what about just C-c C-c on a time.sleep(10) in a fresh session? You can even disable python-mls for this test. You should get something like:

Python 3.10.0 (default, Oct 13 2021, 06:45:00) [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> python.el: native completion setup loaded
>>> import time
>>> time.sleep(10)
  C-c C-cTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyboardInterrupt
>>>
joostkremers commented 2 years ago

Yup, that's exactly what I get (except for the native completion message):

Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.sleep(10)
  C-c C-cTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyboardInterrupt
>>>
jdtsmith commented 2 years ago

OK, it's good news that interruption is working. Let's see if things are being sent correctly through the pre-output filter which hides things behind the scenes:

(defun my/python-mls-preout-log (string)
  (let ((buffer
     (get-buffer-create "*PMLSPO*")))
    (display-buffer buffer)
    (with-current-buffer buffer
      (insert "\nGOT:\n" string))))
(advice-add #'python-shell-output-filter :before #'my/python-mls-preout-log)

And let me know what you see in the *PMLSPO* buffer that appears, when you try a continued command like if True: [Ret]. BTW, you can interrupt a stuck Emacs by sending it a SIGUSR2 signal from the outside.

joostkremers commented 2 years ago

OK, it's good news that interruption is working. Let's see if things are being sent correctly through the pre-output filter which hides things behind the scenes:

The *PMLSPO* buffer shows this a few times:

GOT:

>>>

But those are all added before I hit RET. Once I hit RET after e.g., if True:, nothing is added anymore.

BTW, you can interrupt a stuck Emacs by sending it a SIGUSR2 signal from the outside.

That doesn't seem to work for Windows processes, unfortunately.

BTW, I usually run Python in a virtual env, but just now I accidentally did M-x run-python without activating a venv first (which I do in Emacs with pyvenv-activate) and this warning popped up:

Warning (python): Python shell prompts cannot be detected.
If your emacs session hangs when starting python shells
recover with ‘keyboard-quit’ and then try fixing the
interactive flag for your interpreter by adjusting the
‘python-shell-interpreter-interactive-arg’ or add regexps
matching shell prompts in the directory-local friendly vars:
  + ‘python-shell-prompt-regexp’
  + ‘python-shell-prompt-block-regexp’
  + ‘python-shell-prompt-output-regexp’
Or alternatively in:
  + ‘python-shell-prompt-input-regexps’
  + ‘python-shell-prompt-output-regexps’

This doesn't happen when I first activate a venv and then do M-x run-python, and the problem with python-mls occurs either way, so it's probably not related, but I since it seems to involve the prompt, I thought I'd mention it. (The functionality of the Python shell doesn't appear to be affected by this warning, so I'm not sure what the warning is about. And if that makes me sound like a total noob, then I guess when it comes to Python shells, that's probably true. On Linux, this stuff just seems to work... 😲 )

jdtsmith commented 2 years ago

OK, this doesn't look right. It should mention the interrupt. Something like:

GOT:
KeyboardInterrupt

GOT:
>>> 
GOT:

KeyboardInterrupt

GOT:
>>>

To debug further, please deactivate again the advice for check-prompt as in #7. Then try if True: [Ret]. You should hopefully see .... Does C-c C-c work to return the prompt again in this situation? And with the above log advice still active, also try M-: (python-mls-interrupt-quietly) again and report the new content in the log buffer.

jdtsmith commented 2 years ago

Python shell prompts cannot be detected

You just need python-shell-interpreter-args set to "-i" I bet.

joostkremers commented 2 years ago

OK, this doesn't look right. It should mention the interrupt. Something like:

GOT:
KeyboardInterrupt

Nope, nothing like that.

To debug further, please deactivate again the advice for check-prompt as in #7. Then try if True: [Ret]. You should hopefully see .... Does C-c C-c work to return the prompt again in this situation?

Yes, the ... is visible. Emacs is indeed unresponsive but now I can C-c C-c out of it:

Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> if True:
...   C-c C-c

Traceback (most recent call last):
  File "C:\Users\joost.kremers\.pyenv\pyenv-win\versions\3.9.6\lib\encodings\cp1252.py", line 14, in decode
    def decode(self,input,errors='strict'):
KeyboardInterrupt

The above exception was the direct cause of the following exception:

KeyboardInterrupt: decoding with 'cp1252' codec failed (KeyboardInterrupt: )
>>> >>>

And with the above log advice still active, also try M-: (python-mls-interrupt-quietly) again and report the new content in the log buffer.

Nothing's added to the log buffer if I do that. Here's what I did:

At this point, Emacs is stuck, but I can get back control with C-g. No *PMLSPO* buffer appears.

If I then type something at the prompt, the buffer pops up with:


GOT:
Traceback (most recent call last):
  File "C:\Users\joost.kremers\.pyenv\pyenv-win\versions\3.9.6\lib\encodings\cp1252.py", line 14, in decode
    def decode(self,input,errors='strict'):
KeyboardInterrupt

The above exception was the direct cause of the following exception:

KeyboardInterrupt: decoding with 'cp1252' codec failed (KeyboardInterrupt: )
>>> 
GOT:

>>> 
GOT:

>>> 
GOT:

The error message wasn't there before, BTW...

I tried M-: (python-mls-interrupt-quietly) again, but with the same result (Emacs gets stuck, I need to type C-g and nothing is added to the log buffer.)

jdtsmith commented 2 years ago

Your nested exception and double prompt could be the issue, which isn't really in the domain of python-mls. With check-prompt disabled, on C-c C-c at the ..., I get a simple & clean:

>>> if True:
...   C-c C-c
KeyboardInterrupt
>>>

Please try the first exercise again, after doing M-x trace-function python-shell-comint-end-of-output-p, and report the trace output.

jdtsmith commented 2 years ago

Also, separately, can you try the above, but skipping these steps:

Just start with M-x run-python.

joostkremers commented 2 years ago

Your nested exception and double prompt could be the issue, which isn't really in the domain of python-mls. With check-prompt disabled, on C-c C-c at the ..., I get a simple & clean:

>>> if True:
...   C-c C-c
KeyboardInterrupt
>>>

Tried this again, and it turns out Emacs doesn't become unresponsive at all. For some reason, TAB did not indent, but I can type four spaces and continue inputting code.

Still, doing C-c C-c gives me the exception I pasted above.

Please try the first exercise again,

The first exercise being to disable the advice for comint-output-filter and then tying if True: [RET] in the Python REPL?

after doing M-x trace-function python-shell-comint-end-of-output-p, and report the trace output.

I get:

======================================================================
1 -> (python-shell-comint-end-of-output-p "... ")
1 <- python-shell-comint-end-of-output-p: 0
======================================================================
1 -> (python-shell-comint-end-of-output-p "Traceback (most recent call last):
  File \"C:\\Users\\joost.kremers\\.pyenv\\pyenv-win\\versions\\3.9.6\\lib\\encodings\\cp1252.py\", line 14, in decode
    def decode(self,input,errors='strict'):
KeyboardInterrupt

The above exception was the direct cause of the following exception:

KeyboardInterrupt: decoding with 'cp1252' codec failed (KeyboardInterrupt: )
>>> ")
1 <- python-shell-comint-end-of-output-p: 352
jdtsmith commented 2 years ago

Did you disable the advice to comint-output-filter and C-c C-c? I meant do NOT disable the advice, try an if True: as normal, but with this trace-function. This is the type of thing that would be a 5min debug if I could reproduce.

Emacs doesn't become unresponsive at all

That's expected when you have disabled the advice. To make this clearer, python-mls:

  1. looks for the native "..." continued statement prompt.
  2. Silently interrupts the command underway.
  3. Overwrites the most recent input & output, putting the input back at the prompt.

Your Emacs is hanging at step 2. It is awaiting process output that never arrives. Our job is to figure out why.

joostkremers commented 2 years ago

Did you disable the advice to comint-output-filter and C-c C-c? I meant do NOT disable the advice, try an if True: as normal, but with this trace-function.

Ok, I'll have to post a screen shot, because once I type if True:, Emacs hangs:

image

This is the type of thing that would be a 5min debug if I could reproduce.

Yeah, I'm sorry about all the trouble this is causing... I do have a Windows partition lying around on another laptop that I hardly ever use. I could put Emacs and Python on it and give you remote access, if that would help...

jdtsmith commented 2 years ago

Unfortunately that doesn't include the output from the interruption itself (no ... in the output). If you want to give me remote access I could take a look. Before that though, do try with a normal M-x run-python and no fancy venv. And also perhaps enable both the logging and tracing and repeat the hang to see if that gives any insight.

joostkremers commented 2 years ago

Unfortunately that doesn't include the output from the interruption itself (no ... in the output).

No, but that's because once I hit [RET] after if True:, Emacs hangs. I did try C-c C-c, but there's simply no response.

If you want to give me remote access I could take a look.

I've set up Python and Emacs there with minimal configuration. So if you want to take a look, send me an e-mail. (Address is in my Github profile.)

Before that though, do try with a normal M-x run-python and no fancy venv. And also perhaps enable both the logging and tracing and repeat the hang to see if that gives any insight.

Let me include another screen shot. Here's what I did:

image

jdtsmith commented 2 years ago

Thanks for the very careful description. I’ll email.

jdtsmith commented 2 years ago

FYI, I did email yesterday (in case it landed in your spam box).

jdtsmith commented 2 years ago

@joostkremers let me know how you'd like to proceed.

jdtsmith commented 2 years ago

It seems the key here is python-mls-interrupt-quietly, which is called when the ... continuation prompt is found in the output. To see if Emacs is stuck waiting for process output that never comes, you could see whether the following small change allows you to recover from if True: [Ret].

Alter this:

    (while python-shell-output-filter-in-progress
      (accept-process-output))

to add a 2.5s timeout:

    (while python-shell-output-filter-in-progress
      (accept-process-output nil 2.5))

This isn't a fix, just zeroing in on the issue.

joostkremers commented 2 years ago

No, I'm still not able to recover from if True: [RET], but I do notice that Emacs is no longer consuming 30% of CPU. Instead it's at 0%. I killed and restarted Emacs a few times, so it's consistent, and when I remove the timeout, CPU goes back to 30%.

jdtsmith commented 2 years ago

Wow, it sounds well and truly hung. So perhaps it is just looping over and over receiving no output, with the delay giving it some CPU breathing room.

We can also try limiting the accept loop iterations (again, not a fix):

 (let ((cnt 1)) 
  (while (and (< cnt 10) python-shell-output-filter-in-progress)
    (accept-process-output nil 1)
    (cl-incf cnt)))
joostkremers commented 2 years ago

Yup, that gives me the prompt after waiting about 10 secs. After that, I can continue the if statement normally and execute it, and python-mls then works as it's supposed to:

Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> if True:
    print (1)

1
>>> if True:
    print (2)

2
>>> if True:
    print(3)

3
>>> if True:
    print(4)

4
>>> if True:
    print(4)
    print(5)
else:
    print(6)

4
5
>>>
jdtsmith commented 2 years ago

Very interesting! I presume the (decorative only) leading ... was shown as well, even in the 1st stuck case? So it seems there is an issue with the first interruption (only) producing no output, or perhaps the first interruption's output is getting missed somehow. Of course python-mls is trying to be very careful to wait for that output (a Keyboard Interrupt and a prompt) before allowing you to send another command. So this is the issue.

Can you try your if True: [Ret] C-c in a bare newly run python process in the Windows shell (same interpreter as Emacs would run) and report on what it does? Then try another if True: in the same process. Any difference in output?

joostkremers commented 2 years ago

Very interesting! I presume the (decorative only) leading ... was shown as well,

Right! I didn't even notice it was absent from the output I posted. Yes, the ... is shown on every indented line.

even in the 1st stuck case?

You mean, after I type if True: [RET] and the loop times out? Yes.

Can you try your if True: [Ret] C-c in a bare newly run python process in the Windows shell (same interpreter as Emacs would run) and report on what it does? Then try another if True: in the same process. Any difference in output?

No, no difference.

To make sure we understand each other correctly, though: if True: [RET] hangs Emacs each time if I type it directly. What I meant when I said that "python-mls works as it's supposed to" was that I can hit up arrow and edit the previous input. But if I type if True: [RET] again, Emacs hangs again.

jdtsmith commented 2 years ago

I didn't even notice it was absent from the output I posted.

In fact that's a feature (so the decorative ...'s are removed and reinserted on copy/paste from anywhere).

if True: [RET] hangs Emacs each time if I type it directly.

OK. I had understood you to mean that you could compose multi-line commands as expected after the first times out. I suppose you can also try C-j from the first line instead of [Ret]; that should also work normally.

No, no difference.

And what does that output look like after you C-c a continuing command? BTW, if you can try re-enabling the tracing and logging code we used earlier, and leaving the timeout/loop max count intact, entering a dreaded if True: [Ret], we might in fact get some output to examine after the while loop ends. If you'd prefer to go the remote debug route instead of the back and forth, happy to give that a try.

joostkremers commented 2 years ago

With the logging and the tracing active, the moment the timeout ends, the *trace-output* buffer shows the following:

1 -> (python-shell-comint-end-of-output-p "... ")
1 <- python-shell-comint-end-of-output-p: 0
======================================================================

The *PMLSPO* buffer shows nothing.

When I then start typing, after the third character, the keyboard interrupt happens. The *trace-output* buffer shows this:

1 -> (python-shell-comint-end-of-output-p "Traceback (most recent call last):
  File \"C:\\Users\\joost.kremers\\.pyenv\\pyenv-win\\versions\\3.9.6\\lib\\encodings\\cp1252.py\", line 14, in decode
")
1 <- python-shell-comint-end-of-output-p: nil
======================================================================
1 -> (python-shell-comint-end-of-output-p "Traceback (most recent call last):
  File \"C:\\Users\\joost.kremers\\.pyenv\\pyenv-win\\versions\\3.9.6\\lib\\encodings\\cp1252.py\", line 14, in decode
    def decode(self,input,errors='strict'):
KeyboardInterrupt

The above exception was the direct cause of the following exception:

KeyboardInterrupt: decoding with 'cp1252' codec failed (KeyboardInterrupt: )
>>> ")
1 <- python-shell-comint-end-of-output-p: 352
======================================================================

And the *PMLSPO* buffer this:

GOT:
Traceback (most recent call last):
  File "C:\Users\joost.kremers\.pyenv\pyenv-win\versions\3.9.6\lib\encodings\cp1252.py", line 14, in decode

GOT:
    def decode(self,input,errors='strict'):
KeyboardInterrupt

The above exception was the direct cause of the following exception:

KeyboardInterrupt: decoding with 'cp1252' codec failed (KeyboardInterrupt: )
>>> 
GOT:

>>> 
GOT:

>>>

Not sure if that helps in any way...

jdtsmith commented 2 years ago

OK I got access to a Windows machine with Emacs 27.2 and python v3.10, and could reproduce this. Here is the issue: interrupt-process (an internal C function) does not function correctly to interrupt Python on Windows emacs! While at the command prompt, a simple C-c does correctly interrupt (and returns Keyboard Interrupt\n>>> promptly), interrupt-process does not, and leads to the double-traceback you reported earlier, but only after you send another newline. This is incorrect behavior, and seems to represent a bug in the way Emacs sends interrupts on Windows, which differs from C-c in the Windows command prompt.

It's easy to verify the difference:

So python-mls is waiting "forever" for a prompt which will never show up. This needs to be fixed upstream, so I'd strongly encourage you to report an emacs bug (and maybe test on Emacs 28 first).

In the meantime, as a cheap workaround, you can add:

(process-send-string nil "\n")

to the end of python-mls-interrupt to send a newline and just swallow all the errors. A longer term solution requires fixing the incorrect behavior of Emacs->Python interruption.

joostkremers commented 2 years ago

Thanks for diving into this! Glad you found another Windows machine to debug on... 😄

So python-mls is waiting "forever" for a prompt which will never show up. This needs to be fixed upstream, so I'd strongly encourage you to report an emacs bug (and maybe test on Emacs 28 first).

Will do.

In the meantime, as a cheap workaround, you can add:

(process-send-string nil "\n")

to the end of python-mls-interrupt to send a newline and just swallow all the errors.

Thanks, that works.

A longer term solution requires fixing the incorrect behavior of Emacs->Python interruption.

Which, depending on the nature of the bug, may not happen until Emacs 29, 28 being in pretest now...

jdtsmith commented 2 years ago

Thanks for your help looking into this. Variation in process handling are a common issue I think.

Eli-Zaretskii commented 2 years ago

Emacs on MS-Windows communicates with async sub-processes via pipes, not via PTYs. Does Python on MS-Windows react to the C-c character received from a pipe as it reacts to it when it's attached to a console device? If not, this is the root cause of your problem (although I admit I don't really understand the significance of all the talk about backtraces, and I have no idea what does python-mls do, so I might be missing something).

FTR, Emacs on Windows implements sending SIGINT to a sub-process by setting the foreground window to that of the sub-process, and then simulating the C-c keystroke. IOW, it injects a Ctrl-C character into the input stream of the sub-process. What happens as the result is entirely up to the sub-process, but I could understand why a sub-process whose stdin is a pipe would not react to Ctrl-C as it does when it runs interactively.

jdtsmith commented 2 years ago

Thanks, this is helpful. The simple idea is to notice Python is in the middle of a continued statement (which it indicates with a ... prompt), interrupt that, and recompose using Emacs capabilities. On other systems aside from Windows, interrupt-process immediately returns to the prompt. When running python on Windows from the "Command Prompt", C-c also leads to the correct behavior. But when run as an emacs sub-process, python complains about "decoding errors" (decoding with 'cp1252' codec failed), and does not return to the prompt. I'm not a Windows user, but I wonder if this problem is unique to Python.

I'd speculate, based on this answer, that Python attempts to decode the C-c character over the pipe using the Windows-specific ANSI-like cp1252 codec; not sure why this C-c from Emacs isn't being encoded correctly to 0x03. Perhaps one of the C1 control characters that window-1252 does not define is being sent as well.

Eli-Zaretskii commented 2 years ago

I wonder if this problem is unique to Python.

It isn't.

I'd speculate, based on this answer, that Python attempts to decode the C-c character over the pipe

It could indeed be a side effect of the attempt to decide. At this point, I'd suggest to ask the Python developers, they might know the real answer. Specifically, I'd ask them whether Python is expected to behave like it got SIGINT when it gets Ctrl-C but its stdin is a pipe.

Eli-Zaretskii commented 2 years ago

At this point, I'd suggest to ask the Python developers

It could be that Python closes its window when its stdin is a pipe, in which case the method used by Emacs to deliver SIGINT to it will simply not work on MS-Windows.

jdtsmith commented 2 years ago

Thanks, Eli. @joostkremers would you be willing to put this issue to the python devs? I'm not a Windows users so couldn't offer them much in the way of testing.

joostkremers commented 2 years ago

@joostkremers would you be willing to put this issue to the python devs? I'm not a Windows users so couldn't offer them much in the way of testing.

Yeah, sure. Though I suspect some of the devs will have access to a Windows machine to test on. :smile: