Fontification issue when using with `comint-prompt`

[Yevgnen]

Hi, thanks for the great package.

I'm using the small handmade package comint-prompt to a bit speed up the prompt fontification in comint buffers. I made this because sometimes the (long) output (e.g. running str(list(range(20000))) ) may make the fontification slow and output some immediate result like

The little package a bit speed up this.

However, when I use python-mls with this package, the input and output prompt are not fontified after the first command execution in the *Python* buffer. Since I'm not a font-lock expert, do you have any suggestion on how I/we can fix this? Thanks!

[jdtsmith]

python-MLS speeds up fontification of input (only); it does not fontify the prompt, output or anything else. But neither does inferior-python-mode. What type of fontification are you trying to achieve with your package?

The reason the output here is colored like the prompt is because comint mode is set to recognize anything at the end of the final line with no newline as the prompt, so it is dutifully coloring all this output here in a prompt color, while python slowly/painfully supplies all that text. If you pop to the end of the buffer, you can see text accumulating. C-c C-c will interrupt this, and C-c C-o will flush the last output.

So it's not really a fontification issue; once the final newline and prompt arrive, the incorrect coloring will be taken away. But as you noticed, producing this much output via comint is pretty painful. So maybe there's some way to speed it up; I'll take a look.

[Yevgnen]

What type of fontification are you trying to achieve with your package?

The package simply set comint-highlight-prompt to empty face and use font-lock-add-keywords to fontify the input and output prompts (by (comint-prompt-setup)). This avoids

The reason the output here is colored like the prompt is because comint mode is set to recognize anything at the end of the final line with no newline as the prompt, so it is dutifully coloring all this output here in a prompt color, while python slowly/painfully supplies all that text. If you pop to the end of the buffer, you can see text accumulating. C-c C-c will interrupt this, and C-c C-o will flush the last output. So it's not really a fontification issue; once the final newline and prompt arrive, the incorrect coloring will be taken away. But as you noticed, producing this much output via comint is pretty painful.

and speeds up the output progress of str(list(range(20000)))from 20s to 0.2s on my computer.

I think maybe the font lock regex is broken or overwritten for some reason when using with python-mls. I will do more debugs!

[jdtsmith]

Interesting. Note that neither python-mls nor inferior-python-mode use font-lock mode. Each fontifies input (and only input) "on demand" (with python-mls doing so much more efficiently in-buffer). Comint fontifies other stuff, including prompt.

It's not at all clear to me why setting the face to nil would impact the speed. But I agree it's a problem. Is it unique to python shells in your experience?

[Yevgnen]

Is it unique to python shells in your experience?

It helps other comint buffer too. seq 50000 in a shell buffer takes 8s to output while 2s when with the simple trick. I get my emacs frozen many times by accidentally output large content and the small trick at least gives me a bigger change use C-c C-c to interrupt it.

Note that neither python-mls nor inferior-python-mode use font-lock mode.

Isn't inferior-python-mode using font-lock-mode? It seems to appear in C-h m in emacs -q.

[jdtsmith]

It seems the main issue is that all output is on one line only (print("\n".join(map(str,range(20000)))) is fast). Emacs really struggles with very long lines. I wonder if there's something general that could be fixed in comint to workaround this? so-long mode unfortunately disables too much to make comint usable.

Isn't inferior-python-mode using font-lock-mode? It seems to appear in C-h m in emacs -q.

What I meant is that they don't use it "normally", but instead either force fontify the region after the prompt, or use a round-trip to an external fontification buffer. In any case I think this is a side issue. The real problem here relates to very slow comint output when "all on one line".

[jdtsmith]

If I disable font lock mode, and replace the comint output process filter with a simple:

(defun my/just-insert (proc str)
  (let ((oprocbuf (process-buffer proc)))
    (when (and str oprocbuf (buffer-name oprocbuf))
      (with-current-buffer oprocbuf
    (insert str)))))

it remains equally slow. So clearly not a prompt fontification issue here. BTW, I also don't see a speedup using comint-prompt-setup in the buffer.

[Yevgnen]

I also don't see a speedup using comint-prompt-setup in the buffer.

comint-prompt-setup should be used as global setting before starting the process. But I'm sorry it does not help you too much.

Any way, the original issue is solved by removing the python-mls--fontify-region-function part as I didn't use font-lock (python-shell-font-lock-enable => nil) originally. Thank you for digging into the details!

[jdtsmith]

Interesting. I will look into disabling the fontification if python-shell-font-lock-enable is nil.

BTW, here's a short function that breaks long lines on comint output, and speeds things up for me, from >1min to 5s for your 20000 case. Perhaps you can use something like this and keep font lock if you want:

(defconst my/comint-long-lines-regexp
  (rx (** 120 140 nonl) (any ?\; ? ?: ?,)))

(defun my/comint-filter-break-output-lines (str)
  (when-let ((proc (get-buffer-process (current-buffer)))
         (pmark (process-mark proc)))
    (save-excursion
      (goto-char (- pmark (length str)))
      (forward-line 0)
      (while (and (looking-at my/comint-long-lines-regexp)
          (< (match-end 0) pmark))
    (goto-char (match-end 0))
    (insert "\n")))))

Just (add-hook 'comint-output-filter-functions #'my/comint-filter-break-output-lines).