search

atomontage / xterm-color

ANSI & xterm-256 color text property translator for Emacs
BSD 2-Clause "Simplified" License
214 stars 20 forks source link

Parsing certain terminal codes with xterm-color causes eshell to be slow #27

Closed codesections closed 5 years ago

codesections commented 5 years ago

Thanks so much for this excellent package! It really makes eshell very useable and generally works great!

However, I've encountered a strange issue where parsing certain strings with xterm-color will slow eshell to a crawl (e.g., after parsing the string, there will be a ~1 second lag between pressing a key and having that key show up on the screen). Clearing eshell will resolve the problem. Here is a minimal example of a string that triggers the issue:

[?1h=
───────┬────────────────────────────────────────────────────────────────────────
       │ File: testfile
───────┼────────────────────────────────────────────────────────────────────────
   1   │ hello world
───────┴────────────────────────────────────────────────────────────────────────

[?1l>

(This string was generated using the excellent CLI app bat, which prints files with syntax highlighting. Specifically, I ran bat over a file containing only the text hello world)

I've found a workaround for my particular use case (running bat -n avoids the issue). But I wanted to mention it in case it is causing problems more broadly.

Please let me know if I can do anything else to help diagnose the problem.

atomontage commented 5 years ago

I can't reproduce the issue here. If I just run bat testfile in eshell, then bat refuses to use ANSI escapes. I assume it contains terminal detection code which fails (as it should). I pretty much have to use "bat --color=always testfile" and when I do that it works fine with no slowdown.

In the snippet you posted, there are two ANSI CSI sequences at the beginning and at the end that are not supported by xterm-color because they are terminal commands that set/clear private attributes (DECSET/DECRST). These are skipped by the filtering code. There is also an ANSI sequence that kills characters to the end of the line. This is also not supported and skipped by the processing code. You can try setting your TERM to "eterm" and bat should not use DECSET/DECRST. Either way, I can't see how the slowdown you describe would happen. There's just nothing in xterm-color.el that would cause it. When you press a key, xterm-color.el is not involved in its processing. So I suspect something else is going on, maybe long horizontal lines (which is a known issue for Emacs in general).

Some generic questions:

Are you running latest xterm-color.el? Which version of Emacs/OS? GUI Emacs or CLI Emacs ? Did you byte compile xterm-color.el?

Screenshot attached below.

atomontage commented 5 years ago

bat inside eshell (GUI Emacs 26)

bat

atomontage commented 5 years ago

Some more things for you to try:

In eshell do: setenv TERM eterm In M-x shell: export TERM=eterm

You should get perfect bat output with decorations after this (aligned lines), since the DECSET/DECRST sequences will not be used by bat.

If you're still seeing a slowdown in eshell you might want to change process-connection-type to nil. By default it's using a pty which is going to cause a lot of problems in general. Here is the snippet from my own config:

;; Use pipes to communicate with spawned processes rather than TTYs
;; This solves a number of issues and makes things a lot faster to boot:
;;
;; + eshell pipes work without issue
;; + eshell runs a lot faster
;; + shell runs a lot faster but we lose job control

(setq process-connection-type nil)

;; shell advice so that it executes inside a let-binding that
;; sets process-connection-type to t since running shells without
;; ttys is not very useful.
(defun xristos/advice-shell (f &rest args)
  (let ((process-connection-type t))
    (apply f args)))

(advice-add 'shell :around #'xristos/advice-shell)
codesections commented 5 years ago

Hmmm, none of those steps resolve the issue for me. (If I add in the process-connection-type bit, I get the same result you mentioned: no color from bat unless I set --color=always. But then I get the same slowdown when it prints the sidebar lines).

I'm happy to drop this/close the issue as Could Not Reproduce. I prefer using bat without the lines anyway, so it won't impact me. (And, it's easy enough to clear if it does come up.) I'd only mentioned it in case it's more widespread but, based on your testing, it looks like it could just be a corner case with my config.

atomontage commented 5 years ago

One final thing to try, copy paste the following into a new file, test.el:

(add-to-list 'load-path "path-to-xterm-color-dir") (require 'xterm-color)

(require 'eshell)

(add-hook 'eshell-before-prompt-hook (lambda () (setq xterm-color-preserve-properties t)))

(add-to-list 'eshell-preoutput-filter-functions 'xterm-color-filter) (setq eshell-output-filter-functions (remove 'eshell-handle-ansi-color eshell-output-filter-functions))

Now, start Emacs by running:

emacs -q -l test.el

This will skip your personal configuration. Run M-x eshell and see if the issue happens again.

If it does, let me know what version of Emacs you're running and what OS you're on.

If it doesn't, then something in your config is causing it.

codesections commented 5 years ago

If it doesn't, then something in your config is causing it.

Yep, something in my config is causing it. I'll try selectively disabling lines in my config until I find the culprit and report back with what's causing the issue. Thanks for the help running this down—definitely above and beyond!

codesections commented 5 years ago

Hmm, I've identified the compatibility issue and it's kind of an unfortunate one: I encounter the issue whenever I'm using the smart-shell feature from em-smart. Specifically, I can use bat or other commands just fine if I don't run eshell-smart-initialize, but as so as I run eshell-smart-initialize, I start getting the same input lag I described above.

You've already helped out way more than I could have asked for, but I don't suppose you have any leads about where the incompatibility could be coming from, do you?

I consider the smart shell to be one of the killer features of eshell, so I'll probably stick to the workaround discussed up thread if I can't get the smart shell and xterm-color to play nicely together. I'm guessing (without any basis) that other eshell users might also like the smart shell feature, so this issue might not be 100% idiosyncratic to my setup.

atomontage commented 5 years ago

I still can't reproduce, with em-smart enabled. Try this, in the previous test.el, after (require 'eshell) add the following:

(require 'em-smart) (add-hook 'eshell-mode-hook (lambda () (eshell-smart-initialize)))

Now run emacs again with emacs -q -l test.el, enter eshell and see if it happens. If it doesn't then you need to narrow your config down more.

I'm testing on Emacs 25, Emacs 26 and Emacs 27 and I don't see what you describe.

codesections commented 5 years ago

Thanks. I dug into it a bit but wasn't able to figure out what in my config is causing the issue. I'll keep looking, but probably won't be able to report back for a few days. Thanks again for all your help!

atomontage commented 5 years ago

I'm assuming the issue didn't happen with the updated test.el thus it's something else in your config that triggers it (possibly interacting with em-shell). I'm closing this as I can't reproduce but if you do manage to find the part of your config that triggers the issue, let me know and I'll have another look.

codesections commented 5 years ago

I have identified the problem, and it was in one of the last places I would have expected problems with my config. The problem comes from using certain fonts with both xterm-color and eshell. Here is a minimum test.el config that reproduces the issue:

(add-to-list 'load-path "~/.emacs.d/elpa/xterm-color-20190315.35")
(require 'xterm-color)

(require 'eshell)

(add-hook 'eshell-before-prompt-hook
          (lambda ()
            (setq xterm-color-preserve-properties t)))

(add-to-list 'eshell-preoutput-filter-functions 'xterm-color-filter)
(setq eshell-output-filter-functions
      (remove 'eshell-handle-ansi-color eshell-output-filter-functions))

(require 'em-smart)
(add-hook 'eshell-mode-hook (lambda () (eshell-smart-initialize)))
(add-to-list 'default-frame-alist
            '(font . "DejaVu Sans Mono-12"))

(setenv "PAGER" "")
(setenv "TERM" "eterm")

I showed the issue with DejaVu Sans Mono, since I know that to be a popular font for Emacs. I was originally using Hack; the issue is the same for both fonts. It does not occur with all fonts, however; the first font I found without the issue was ubunu mono. I haven't yet figured out what the commonality for fonts causing the issue is.

This turned out to be one of the odder bugs I've encountered. Thanks again for all your help!

(Oh, and this was on Emacs 26.1, since you'd asked about that earlier.)

atomontage commented 5 years ago

This is bizarre, I still can't reproduce your issue. I tried DejaVu Sans Mono and Hack without any problems.

Emacs 25, 26 and 27 compiled from git master.

The fact that it's triggered by a font?!? makes it extremely likely that it's a bug in Emacs itself, possibly a bug in the combination of Emacs/backend/display system.

codesections commented 5 years ago

Totally agreed that it's bizarre! But happy to leave this closed as can't reproduce and change fonts. (I wasn't that tied to Hack anyway).

Thanks again for all your help. I definitely owe you one