racket-repl-mode ansi escape code coloring

[tgbugs]

Before the changes to repl-mode in 25224889d20b37bfd0d315a656542bb4fe8c2076 ansi escape codes were processed correctly. After the changes they are no longer processed correctly. Is there some way to get this working again?

Steps to reproduce: In the repl run

(system "echo $(tput setaf 4)should be blue$(tput sgr0)")

The output should be blue as it was in the comint version of the repl and as it would be if running racket from the command line.

The actual result in some cases contains raw escape charachters, possibly when logging to stderr instead of stdout?

[2023-12-08 12:58:41,141] -  WARNING -        sparcur - fetch_metadata_files.py:45   - No paths to fetch, did you pull the file system metadata?

Package

metadata

nil

package-archives

(("gnu" . "https://elpa.gnu.org/packages/")
 ("nongnu" . "https://elpa.nongnu.org/nongnu/")
 ("melpa" . "https://melpa.org/packages/"))

racket--el-source-dir

"/home/tom/.emacs.d/lisp/racket-mode/"

racket--rkt-source-dir

"/home/tom/.emacs.d/lisp/racket-mode/racket/"

System values

emacs-version

"29.1"

major-mode

help-mode

system-type

gnu/linux

x-gtk-use-system-tooltips

t

display-graphic-p

t

Buffer values

after-change-functions

nil

before-change-functions

nil

completion-at-point-functions

(tags-completion-at-point-function)

eldoc-documentation-function

eldoc-documentation-default

font-lock-defaults

nil

pre-command-hook

(evil--jump-hook t)

post-command-hook

(evil--jump-handle-buffer-crossing t)

post-self-insert-hook

(electric-pair-post-self-insert-function electric-pair-open-newline-between-pairs-psif electric-indent-post-self-insert-function blink-paren-post-self-insert-function)

xref-backend-functions

(etags--xref-backend)

Racket Mode values

racket--cmd-open-p

t

racket-after-run-hook

nil

racket-back-end-configurations

((:directory "/" :racket-program nil :remote-source-dir nil :restart-watch-directories nil :windows nil))

racket-before-run-hook

nil

racket-browse-url-function

racket-browse-url-using-temporary-file

racket-command-timeout

10

racket-documentation-search-location

"https://docs.racket-lang.org/search/index.html?q=%s"

racket-error-context

medium

racket-hash-lang-mode-hook

nil

racket-hash-lang-module-language-hook

nil

racket-hash-lang-token-face-alist

((constant . font-lock-constant-face)
 (error . error)
 (other . font-lock-doc-face)
 (keyword . font-lock-keyword-face)
 (hash-colon-keyword . racket-keyword-argument-face)
 (at . font-lock-doc-face))

racket-history-filter-regexp

"\\`\\s *\\'"

racket-imagemagick-props

nil

racket-images-inline

t

racket-images-keep-last

100

racket-images-system-viewer

"display"

racket-indent-curly-as-sequence

t

racket-indent-sequence-depth

0

racket-logger-config

((cm-accomplice . warning)
 (GC . info)
 (module-prefetch . warning)
 (optimizer . info)
 (racket/contract . error)
 (racket-mode-debugger . info)
 (sequence-specialization . info)
 (* . fatal))

racket-memory-limit

2048

racket-mode-hook

(rainbow-delimiters-mode tgbugs-racket-hs racket-xp-mode rcc goto-address-mode tgbugs-buffer-local-tab-complete tgbugs-racket-outline-helper electric-indent-local-mode)

racket-module-forms

"\\s(\\(?:module[*+]?\\|library\\)"

racket-pretty-lambda

nil

racket-pretty-print

t

racket-program

"racket"

racket-repl-buffer-name-function

nil

racket-repl-history-directory

"~/.emacs.d/racket-mode/"

racket-repl-mode-hook

(rainbow-delimiters-mode tgbugs-buffer-local-tab-complete)

racket-sexp-comment-fade

0.5

racket-shell-or-terminal-function

racket-shell

racket-show-functions

(racket-show-echo-area)

racket-smart-open-bracket-enable

nil

racket-submodules-to-run

((test)
 (main))

racket-use-repl-submit-predicate

nil

racket-xp-add-binding-faces

nil

racket-xp-after-change-refresh-delay

1

racket-xp-highlight-unused-regexp

"^[^_]"

racket-xp-mode-lighter

(:eval
 (racket--xp-mode-lighter))

Minor modes

enabled

((async-bytecomp-package-mode)
 (auto-composition-mode)
 (auto-compression-mode)
 (auto-encryption-mode)
 (buffer-read-only)
 (column-number-mode)
 (csv-field-index-mode)
 (display-line-numbers-mode)
 (electric-pair-mode)
 (evil-collection-unimpaired-mode)
 (evil-local-mode)
 (evil-mode)
 (file-name-shadow-mode)
 (font-lock-mode)
 (global-display-line-numbers-mode)
 (global-eldoc-mode)
 (global-evil-collection-unimpaired-mode)
 (global-font-lock-mode)
 (global-git-commit-mode)
 (global-undo-tree-mode)
 (isearch-fold-quotes-mode)
 (line-number-mode)
 (magit-auto-revert-mode)
 (mouse-wheel-mode)
 (override-global-mode)
 (savehist-mode)
 (semantic-minor-modes-format)
 (shell-dirtrack-mode)
 (tooltip-mode)
 (transient-mark-mode)
 (treemacs-filewatch-mode)
 (treemacs-follow-mode)
 (treemacs-fringe-indicator-mode)
 (treemacs-git-mode)
 (undo-tree-mode)
 (url-handler-mode)
 (which-function-mode))

Disabled minor modes

disabled

((abbrev-mode)
 (archive-subfile-mode)
 (auto-complete-mode)
 (auto-fill-function)
 (auto-fill-mode)
 (auto-revert-mode)
 (auto-revert-tail-mode)
 (auto-save-mode)
 (auto-save-visited-mode)
 (avy-linum-mode)
 (bibtex-completion-notes-global-mode)
 (bibtex-completion-notes-mode)
 (blink-cursor-mode)
 (buffer-face-mode)
 (bug-reference-mode)
 (bug-reference-prog-mode)
 (button-mode)
 (cider--debug-mode)
 (cider-auto-test-mode)
 (cider-enlighten-mode)
 (cider-mode)
 (cider-popup-buffer-mode)
 (cl-old-struct-compat-mode)
 (comint-fontify-input-mode)
 (compilation-minor-mode)
 (compilation-shell-minor-mode)
 (completion-in-region-mode)
 (context-menu-mode)
 (crdt-mode)
 (crdt-org-sync-overlay-mode)
 (crdt-visualize-author-mode)
 (csv-align-mode)
 (cursor-face-highlight-mode)
 (cursor-intangible-mode)
 (cursor-sensor-mode)
 (dap-auto-configure-mode)
 (dap-mode)
 (dash-fontify-mode)
 (defining-kbd-macro)
 (delete-selection-mode)
 (diff-auto-refine-mode)
 (diff-minor-mode)
 (dired-hide-details-mode)
 (dired-isearch-filenames-mode)
 (edebug-backtrace-mode)
 (edebug-mode)
 (edit-indirect--overlay)
 (eldoc-mode)
 (electric-indent-mode)
 (electric-layout-mode)
 (electric-quote-mode)
 (elpy-django)
 (elpy-mode)
 (eshell-arg-mode)
 (eshell-command-mode)
 (eshell-proc-mode)
 (eshell-var-mode)
 (ess-elisp-trace-mode)
 (eval-sexp-fu-flash-mode)
 (evil-cleverparens-mode)
 (evil-collection-magit-toggle-text-minor-mode)
 (evil-leader-mode)
 (evil-org-mode)
 (evil-paredit-mode)
 (evil-surround-mode)
 (flycheck-mode)
 (flymake-mode)
 (flyspell-mode)
 (gcmh-mode)
 (geiser-autodoc-mode)
 (geiser-mode)
 (geiser-repl-autoeval-mode)
 (geiser-smart-tab-mode)
 (git-commit-mode)
 (global-auto-complete-mode)
 (global-auto-revert-mode)
 (global-dash-fontify-mode)
 (global-evil-leader-mode)
 (global-evil-surround-mode)
 (global-flycheck-mode)
 (global-goto-address-mode)
 (global-hl-line-mode)
 (global-hl-todo-mode)
 (global-linum-mode)
 (global-prettify-symbols-mode)
 (global-reveal-mode)
 (global-semantic-highlight-edits-mode)
 (global-semantic-highlight-func-mode)
 (global-semantic-show-parser-state-mode)
 (global-semantic-show-unmatched-syntax-mode)
 (global-semantic-stickyfunc-mode)
 (global-tree-sitter-mode)
 (global-visual-line-mode)
 (gnus-dead-summary-mode)
 (gnus-undo-mode)
 (goto-address-mode)
 (goto-address-prog-mode)
 (haskell-indentation-mode)
 (header-line-indent-mode)
 (helm--minor-mode)
 (helm--remap-mouse-mode)
 (helm-autoresize-mode)
 (helm-display-line-numbers-mode)
 (helm-migemo-mode)
 (helm-popup-tip-mode)
 (highlight-numbers-mode)
 (hl-line-mode)
 (hl-todo-mode)
 (horizontal-scroll-bar-mode)
 (hs-minor-mode)
 (ido-everywhere)
 (indent-tabs-mode)
 (isearch-mode)
 (ispell-minor-mode)
 (jit-lock-debug-mode)
 (julia-repl-mode)
 (jupyter-org-interaction-mode)
 (jupyter-repl-interaction-mode)
 (jupyter-repl-persistent-mode)
 (linum-mode)
 (lispy-goto-mode)
 (lispy-mode)
 (lispy-other-mode)
 (lock-file-mode)
 (lost-selection-mode)
 (lsp-inlay-hints-mode)
 (lsp-installation-buffer-mode)
 (lsp-java-lens-mode)
 (lsp-managed-mode)
 (lsp-mode)
 (lsp-signature-mode)
 (lsp-treemacs-deps-list-mode)
 (lsp-treemacs-error-list-mode)
 (lsp-treemacs-generic-mode)
 (lsp-treemacs-sync-mode)
 (macrostep-mode)
 (magit-blame-mode)
 (magit-blame-read-only-mode)
 (magit-blob-mode)
 (magit-wip-after-apply-mode)
 (magit-wip-after-save-local-mode)
 (magit-wip-after-save-mode)
 (magit-wip-before-change-mode)
 (magit-wip-initial-backup-mode)
 (magit-wip-mode)
 (mail-abbrevs-mode)
 (markdown-live-preview-mode)
 (menu-bar-mode)
 (mml-mode)
 (next-error-follow-minor-mode)
 (org-capture-mode)
 (org-cdlatex-mode)
 (org-list-checkbox-radio-mode)
 (org-make-toc-mode)
 (org-src-mode)
 (org-table-follow-field-mode)
 (org-table-header-line-mode)
 (orgstrap-edit-mode)
 (orgtbl-mode)
 (outline-minor-mode)
 (overwrite-mode)
 (paragraph-indent-minor-mode)
 (paredit-mode)
 (polymode-minor-mode)
 (prettify-symbols-mode)
 (pycoverage-mode)
 (pyvenv-mode)
 (pyvenv-tracking-mode)
 (racket-hash-lang-repl-mode)
 (racket-smart-open-bracket-mode)
 (racket-xp-mode)
 (rainbow-delimiters-mode)
 (rainbow-delimiters-org-mode)
 (read-extended-command-mode)
 (rectangle-mark-mode)
 (reveal-mode)
 (rst-minor-mode)
 (semantic-highlight-edits-mode)
 (semantic-highlight-func-mode)
 (semantic-mode)
 (semantic-show-parser-state-mode)
 (semantic-show-unmatched-syntax-mode)
 (semantic-stickyfunc-mode)
 (server-mode)
 (sh-electric-here-document-mode)
 (shell-command-with-editor-mode)
 (shell-highlight-undef-mode)
 (show-paren-mode)
 (show-smartparens-global-mode)
 (show-smartparens-mode)
 (size-indication-mode)
 (slime-autodoc-mode)
 (slime-edit-value-mode)
 (slime-editing-mode)
 (slime-fuzzy-target-buffer-completions-mode)
 (slime-macroexpansion-minor-mode)
 (slime-mode)
 (slime-popup-buffer-mode)
 (slime-repl-map-mode)
 (slime-repl-read-mode)
 (slime-trace-dialog-autofollow-mode)
 (slime-trace-dialog-hide-details-mode)
 (slime-trace-dialog-minor-mode)
 (smartparens-global-mode)
 (smartparens-global-strict-mode)
 (smartparens-mode)
 (smartparens-strict-mode)
 (smerge-mode)
 (symex-editing-mode)
 (symex-mode)
 (tab-bar-history-mode)
 (tab-bar-mode)
 (table-fixed-width-mode)
 (table-mode-indicator)
 (tablist-edit-column-minor-mode)
 (tablist-minor-mode)
 (tar-subfile-mode)
 (temp-buffer-resize-mode)
 (text-scale-mode)
 (tool-bar-mode)
 (transient-resume-mode)
 (tree-sitter-hl-mode)
 (tree-sitter-mode)
 (treemacs-hide-gitignored-files-mode)
 (treemacs-indent-guide-mode)
 (treemacs-indicate-top-scroll-mode)
 (treesit-explore-mode)
 (treesit-inspect-mode)
 (undelete-frame-mode)
 (undo-tree-visualizer-selection-mode)
 (use-hard-newlines)
 (vc-dir-git-mode)
 (vc-parent-buffer)
 (vdiff-3way-mode)
 (vdiff-mode)
 (vdiff-scroll-lock-mode)
 (view-mode)
 (visible-mode)
 (visual-line-mode)
 (vterm-copy-mode)
 (window-divider-mode)
 (with-editor-mode)
 (xref-etags-mode)
 (yas-global-mode)
 (yas-minor-mode))

[greghendershott]

Thanks for the report. I wrongly assumed that was one of the many comint things that were for use by things like M-x shell. I didn't realize people would do that from Racket output.

comint did this by calling ansi-color-process-output on all output before inserting into the buffer.

I could pretty simply revive that same indiscriminate behavior, where an escape affects everything in the buffer.

But:

A main motivation for the new REPL is to use structured output. (Typically, many kinds of output get dumped into the same text stream; then you (try to) recover them via regexps. Whereas with the new REPL, the back end preserves the output structure for the front end REPL.)

And so what I'd want to do is revive this more smartly -- to have the escapes affect only portions of the REPL that are stdout or stderr. For example a "blue" escape should affect the color subsequent stdout/sterr output regions in the buffer -- but not other regions, like error messages, prompts, values, file locations, etc.

Doing this would be more work, but not terribly difficult, from a quick glance at ansi-color-apply-on-region. (I think I can use its ansi-color-context-region state variable to preserve the codes state, but change the position marker to be the next stdout/stderr region. Or something like that.)

I'll take a look.

[greghendershott]

From a quick experiment, that seems to work well.

Something like:

> (display "\033[30;47mblah blah blah")
blah blah blah
> (display "foo")
foo

Causes the "blah blah blah" and "foo" regions to be colored -- but not the following prompt and new input. Just as I want and would expect, in the new REPL.

Also I reset any escape code start upon the next racket-run, which IIUC is correct.

Whereas with M-x shell and running racket there, everything is colored, even the prompt and input. And the escape code state is stuck in the buffer until an explicit reset. Which I don't want. (And IIRC is how the old REPL would have done all this.)

So, after sleeping on it and reviewing, I might have a commit to merge tomorrow.

---

But one thing that confuses me:

Your example using system and tput works for me running Racket in Gnome terminal, outside Emacs.

But it does not give me any blue color in Emacs. Not in M-x shell. Not in my just-updated Racket REPL.

I don't see blue and I don't see any raw codes.

It seems like maybe tput isn't emitting the escape codes for me running something via Emacs?

I am pretty ignorant about escape codes and terminals. Is this because when the emacs process is running racket, tput doesn't see it as a certain kind of terminal?? (For me. But presumably you have it configured differently?)

[greghendershott]

Oh maybe it's because TERM env var for me is xterm-256color in Gnome terminal but dumb in Emacs?

Presumably if someone like you has set TERM as you want for Emacs, then when I launch the racket process from Emacs it will inherit that, and doing (system " ... tput ... ") will do what you want?

[greghendershott]

In an M-x shell buffer, tput setaf works when I use -T xterm-256color ("don't look at TERM, just produce SGR escapes"):

$ echo $(tput -T xterm-256color setaf 4)should be blue$(tput -T xterm-256color setaf 0)
should be blue
$ racket
Welcome to Racket v8.10.0.3 [cs].
> (require racket/system)
> (system "echo $(tput -T xterm-256color setaf 4)should be blue$(tput -T xterm-256color setaf 0)")
should be blue
#t

[greghendershott]

I just pushed a commit (not yet merged to the main branch) that handles SGR escapes to colorize. By default; it can also be configured to show them raw or filter them with no colorizing.

---

p.s. Although I don't understand exactly how/when tput emits SGR escapes or not, in Emacs generally, I think that's a general Emacs configuration question.

If you have any tips in this regard I'd be happy to add them to the docs for racket-repl-mode (but AFAICT it's not something racket-repl-mode can or should solve?).

[greghendershott]

I went ahead and merged this. I think the TL;DR is, if you have things configured in such a way that tput emits SGR escapes when run from Emacs, and you see colors in e.g. shell-mode (which uses comint-mode)... then you should now see colors in racket-repl-mode.

If not, or if you have any other questions or comments of course please let me know.