Strange interaction with `company-mode` on Windows.

[joostkremers]

Hi, this is really a great package and it's working perfectly on Linux, but I'm running into a strange problem on Windows involving company-mode. When python-mls-mode is active in an inferior Python buffer, whenever completion is triggered, point jumps to the beginning of the line (to the first character of the input, to be exact).

It doesn't matter if completion is triggered automatically (which happens after typing three characters) or if I do M-x company-complete RET. I get the same effect in both cases.

Like I said, this only happens on Windows. On Linux, where I use the same setup, everything works fine.

If I disable python-mls (comment it out in my init file and restart Emacs), the problem disappears.

Any idea what might be going on here? Or any tips where I could start debugging this?

[jdtsmith]

Very strange. I haven't seen this on MacOS either. Beginning of line or all the way back to the prompt (on multi-line commands)? Python or iPython? If you disable company mode and just use old fashioned completion, does that result in the same issue?

My strong hunch is that this relates to how inferior-python-shell does completion, sending commands "behind the scenes" and hiding the output. It's all a bit ugly. Perhaps on Windows that output is coming from the Python process chunked in a different manner. Python-mls doesn't do anything related to completion. It does add something to comint-output-filter to check for prompts; that could be the issue but it shouldn't run except when sending commands. You could M-x trace-function python-mls-check-prompt and double-check this. Or even (in the shell buffer) M-: (remove-hook 'comint-output-filter-functions #'python-mls-check-prompt) to remove the prompt checking and see if that changes anything.

[joostkremers]

Very strange. I haven't seen this on MacOS either. Beginning of line or all the way back to the prompt (on multi-line commands)?

I haven't even gotten to trying multiline prompt, what with point jumping after every character I type...

So what I see is this. I start an inferior Python process, then start typing at the prompt. The moment I type the third character, point jumps to the position indicated by the pipe bar:

Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> |imp

Python or iPython?

Python.

If you disable company mode and just use old fashioned completion, does that result in the same issue?

Not sure how to use old fashioned completion... On Linux, completion actually works (I type imp and I see import as a candidate, which I can select with RET). On my Windows setup, there are no completion candidates, even after disabling python-mls. (I'm still trying to figure out why...)

My strong hunch is that this relates to how inferior-python-shell does completion, sending commands "behind the scenes" and hiding the output. It's all a bit ugly. Perhaps on Windows that output is coming from the Python process chunked in a different manner. Python-mls doesn't do anything related to completion. It does add something to comint-output-filter to check for prompts; that could be the issue but it shouldn't run except when sending commands. You could M-x trace-function python-mls-check-prompt and double-check this. Or even (in the shell buffer) M-: (remove-hook 'comint-output-filter-functions #'python-mls-check-prompt) to remove the prompt checking and see if that changes anything.

I'll try those things as soon as I have some time to experiment. (Which, unfortunately, isn't today...)

Thanks!

[jdtsmith]

In the meantime I have pushed some changes which affect how check-prompt works (not for this issue). Worth trying the most recent in case anything has changed.

If not, try these and get back to me. To disable fancy completion, just M-x company-mode and M-Tab to complete.

[joostkremers]

(on multi-line commands)

When I disable company-mode I'm of course able to type more than just three characters, so I tried entering a multi-line command, but as soon as I hit RET after the first line, Emacs hangs and I need to kill the process.

This may be related, or it may not be, but I thought it'd be better to mention that here first, before starting a separate issue.

In the meantime I have pushed some changes which affect how check-prompt works (not for this issue). Worth trying the most recent in case anything has changed.

Unfortunately, this hasn't changed anything.

If not, try these and get back to me. To disable fancy completion, just M-x company-mode and M-Tab to complete.

Heh, 20+ years of Emacs use and I finally realise that standard completion is not automatic. :smile:

Anyway, with company-mode disabled, the problem still occurs when I do ESC TAB or C-M-i (M-TAB is used by Windows). So at least that means company-mode is not involved.

Tracing python-mls-check-prompt shows the function being called when completion is triggered:

======================================================================
1 -> (python-mls-check-prompt #<process Python> "
>>> ")
1 <- python-mls-check-prompt: nil
======================================================================
1 -> (python-mls-check-prompt #<process Python> "
>>> ")
1 <- python-mls-check-prompt: nil
======================================================================
1 -> (python-mls-check-prompt #<process Python> "
>>> ")
1 <- python-mls-check-prompt: nil

The function python-mls-check-prompt isn't in the buffer-local value of comint-output-filter-functions however, so removing it has no effect:

Elisp> (buffer-local-value 'comint-output-filter-functions (get-buffer "*Python*"))
(python-pdbtrack-comint-output-filter-function ansi-color-process-output python-shell-comint-watch-for-first-prompt-output-filter python-comint-postoutput-scroll-to-bottom comint-watch-for-password-prompt)

This contrasts with Linux, BTW, where everything works andpython-mls-check-prompt gets added to comint-output-filter-functions.

Another contrast between Windows and Linux: On Linux, no advice is added to comint-output-filter, while on Windows, it's advised with two functions: python-mls-check-prompt and python-mls--comint-output-filter-fix-rear-nonsticky. Since I'm using Emacs 27 on Window vs. Emacs 28 on Linux, the absence of the second advice makes sense, but I'm not sure why python-ml-check-prompt isn't advising comint-output-filter. (Though it doesn't seem to cause any trouble.)

[jdtsmith]

Sorry this confusion on output-filter was my fault: the new version forgoes that method, and instead adds as advice. So:

(advice-remove #'comint-output-filter #'python-mls-check-prompt)

is what you want to try. But honestly I doubt this is it.

with company-mode disabled, the problem still occurs when I do ESC TAB or C-M-i

Do you see any output or a Completions buffer, or does point just go back to the prompt?

I have a hard time understanding what part of python-mls completion at point could be calling or interacting with.

Here's something else to try: put some text at the prompt, then M-: (python-shell-send-string-no-output "print('test')") and see what happens. For me, this goes back to the beginning of prompt, just like for you (with or without python-mls enabled).

Do you have have "native completions" enabled on both your Linux and Windows setups? For me, only with "native completions" does it behave correctly.

[jdtsmith]

When I disable company-mode I'm of course able to type more than just three characters, so I tried entering a multi-line command, but as soon as I hit RET after the first line, Emacs hangs and I need to kill the process.

This may be related, or it may not be, but I thought it'd be better to mention that here first, before starting a separate issue.

Sounds like a different and more serious issue; can you open it separately?

[joostkremers]

Sorry this confusion on output-filter was my fault: the new version forgoes that method, and instead adds as advice. So:

(advice-remove #'comint-output-filter #'python-mls-check-prompt)

is what you want to try. But honestly I doubt this is it.

If I remove the advice, point stops jumping...

with company-mode disabled, the problem still occurs when I do ESC TAB or C-M-i

Do you see any output or a Completions buffer, or does point just go back to the prompt?

Point just jumps. There is no completion.

Here's something else to try: put some text at the prompt, then M-: (python-shell-send-string-no-output "print('test')") and see what happens. For me, this goes back to the beginning of prompt, just like for you (with or without python-mls enabled).

It goes back to the prompt if I have python-mls enabled, but when I disable python-mls (in my init file, then restart Emacs), I get a debugger buffer:

Debugger entered--Lisp error: (wrong-type-argument arrayp nil)
  substring(nil 0 0)
  replace-regexp-in-string("\\`\\^" "" nil)
  python-shell-comint-end-of-output-p("test\n>>> ")
  python-shell-output-filter("test\n>>> ")
  #f(compiled-function (process string) #<bytecode 0x1f904fd>)(#<process Python> "test\n>>> ")
  apply(#f(compiled-function (process string) #<bytecode 0x1f904fd>) (#<process Python> "test\n>>> "))
  comint-output-filter(#<process Python> "test\n>>> ")
  accept-process-output(#<process Python>)
  python-shell-send-string-no-output("print('test')")
  eval((python-shell-send-string-no-output "print('test')") t)
  eval-expression((python-shell-send-string-no-output "print('test')") nil nil 127)
  funcall-interactively(eval-expression (python-shell-send-string-no-output "print('test')") nil nil 127)
  call-interactively(eval-expression nil nil)
  command-execute(eval-expression)

Do you have have "native completions" enabled on both your Linux and Windows setups? For me, only with "native completions" does it behave correctly.

Oh, there's a difference between Linux and Windows I hadn't realized. On Linux it's enabled, on Windows it's not. Native completion is disabled automatically when opening a Python shell on Windows. From what I understand, native completion requires readline, but readline isn't available on Windows. (There seems to be an alternative for readline on Windows, but I haven't tried it yet to see if it can be used to provide completion.)

[jdtsmith]

OK, I have pushed a fix for a small bug which was allowing the prompt checker to run during completion. Apparently this is only a problem for the non-native completion, which is why you discovered it on Windows. Please give a test.

[joostkremers]

Well, that seems to have fixed the problem of point jumping. Thanks! 🙂

Unfortunately, Emacs still hangs when I hit RET after a line that expects an indented block, so I'll open a new issue.