Extra newlines cause Rhombus to send EOF and exit/disconnect REPL

[countvajhula]

I'm trying out Rhombus, and am attempting to send lines of code for evaluation as regions (via racket-eval-region). This works with normal Racket code using a standard 8.3 REPL. But I've configured .rhm files at a .../rhombus/* path to use a bleeding edge Racket repo binary as backend. Doing C-c C-r on a region works the first time, and then repeating C-c C-r again causes the REPL to crash:

Process *Racket REPL <~/work/racket/rhombus>* connection broken by remote peer

Steps to reproduce:

[Note: if you're running Racket 8.5, it may be worth trying this on a normal Racket buffer first, as it may not be specific to Rhombus (see below for more details)] [Note 2: If it does work fine for Racket 8.5 and you don't have Rhombus installed, the installation steps I followed are: (1) clone the racket/racket repo. (2) Run make, (3) [after a couple of hours] bin/raco pkg install rhombus-prototype, (4) in Racket Mode, use the bin/racket binary as the backend for some/path/rhombus, (5) continue with the steps below]

1. Create a file test.rhm containing:

   
   #lang rhombus

1+4

2. `racket-run` the buffer
3. Select the line `1+4` and move point down to the next line so that the entire line is selected _including the newline_.
4. `C-c C-r` (`racket-send-region`)
5. `C-c C-r` again.

The error also happens if you attempt to evaluate a completely blank line (including the newline). With my default Racket 8.3 backend and a Racket source file, doing this just causes a newline to be inserted in the REPL buffer (i.e. no crash).

I've noticed that in a Racket REPL in a terminal (non-Emacs) shell, hitting Enter inserts a newline. On the other hand, in the latest 8.5 REPL, hitting Enter does nothing. It seems like this might have something to do with it, and I'm not sure if this behavior is unique to the bleeding edge version or if newer-than-8.3 REPLs do already have this behavior (in which case, the problem may not be Rhombus-specific).

<details>
<pre>
((alist-get 'racket-mode package-alist))
((emacs-version "27.1")
 (system-type darwin)
 (x-gtk-use-system-tooltips UNDEFINED)
 (major-mode racket-mode)
 (racket--el-source-dir "/Users/siddhartha/.emacs.d/straight/build/racket-mode/")
 (racket--rkt-source-dir "/Users/siddhartha/.emacs.d/straight/build/racket-mode/racket/")
 (racket-program "racket")
 (racket-command-timeout 10)
 (racket-path-from-emacs-to-racket-function UNDEFINED)
 (racket-path-from-racket-to-emacs-function UNDEFINED)
 (racket-browse-url-function racket-browse-url-using-temporary-file)
 (racket-documentation-search-location "https://docs.racket-lang.org/search/index.html?q=%s")
 (racket-xp-after-change-refresh-delay 1)
 (racket-xp-mode-lighter
  (:eval
   (racket--xp-mode-lighter)))
 (racket-xp-highlight-unused-regexp "^[^_]")
 (racket-repl-buffer-name-function nil)
 (racket-submodules-to-run
  ((test)
   (main)))
 (racket-memory-limit 2048)
 (racket-error-context medium)
 (racket-repl-history-directory "~/.emacs.d/racket-mode/")
 (racket-history-filter-regexp "\\`\\s *\\'")
 (racket-images-inline t)
 (racket-imagemagick-props nil)
 (racket-images-keep-last 100)
 (racket-images-system-viewer "open")
 (racket-pretty-print t)
 (racket-use-repl-submit-predicate nil)
 (racket-pretty-print t)
 (racket-indent-curly-as-sequence t)
 (racket-indent-sequence-depth 0)
 (racket-pretty-lambda nil)
 (racket-smart-open-bracket-enable nil)
 (racket-module-forms "\\s(\\(?:module[*+]?\\|library\\)")
 (racket-logger-config
  ((cm-accomplice . warning)
   (GC . info)
   (module-prefetch . warning)
   (optimizer . info)
   (racket/contract . error)
   (racket-mode-debugger . info)
   (sequence-specialization . info)
   (* . fatal)))
 (racket-show-functions
  (racket-show-pseudo-tooltip)))
(enabled-minor-modes
 (auto-composition-mode)
 (auto-compression-mode)
 (auto-encryption-mode)
 (auto-fill-mode)
 (auto-save-mode)
 (blink-cursor-mode)
 (buffer-ring-mode)
 (column-number-mode)
 (company-box-mode)
 (company-mode)
 (company-prescient-mode)
 (counsel-mode)
 (display-line-numbers-mode)
 (eldoc-mode)
 (electric-indent-mode)
 (evil-collection-unimpaired-mode)
 (evil-commentary-mode)
 (evil-goggles-mode)
 (evil-local-mode)
 (evil-matchit-mode)
 (evil-mc-mode)
 (evil-mode)
 (evil-surround-mode)
 (file-name-shadow-mode)
 (font-lock-mode)
 (general-override-mode)
 (global-auto-revert-mode)
 (global-company-mode)
 (global-display-line-numbers-mode)
 (global-eldoc-mode)
 (global-evil-collection-unimpaired-mode)
 (global-evil-matchit-mode)
 (global-evil-mc-mode)
 (global-evil-surround-mode)
 (global-font-lock-mode)
 (global-git-commit-mode)
 (global-tree-sitter-mode)
 (global-undo-tree-mode)
 (global-visual-line-mode)
 (global-whitespace-mode)
 (hs-minor-mode)
 (ivy-mode)
 (ivy-prescient-mode)
 (ivy-rich-mode)
 (mac-mouse-wheel-mode)
 (magit-auto-revert-mode)
 (menu-bar-mode)
 (override-global-mode)
 (prescient-persist-mode)
 (projectile-mode)
 (racket-xp-mode)
 (recentf-mode)
 (semantic-minor-modes-format)
 (shell-dirtrack-mode)
 (show-paren-mode)
 (straight-package-neutering-mode)
 (straight-use-package-mode)
 (symex-mode)
 (telephone-line-mode)
 (tooltip-mode)
 (transient-mark-mode)
 (undo-tree-mode)
 (visual-line-mode)
 (winner-mode)
 (yas-global-mode)
 (yas-minor-mode))
(disabled-minor-modes
 (abbrev-mode)
 (ace-window-display-mode)
 (ace-window-mode)
 (adaptive-wrap-prefix-mode)
 (all-the-icons-dired-mode)
 (archive-subfile-mode)
 (auto-fill-function)
 (auto-revert-mode)
 (auto-revert-tail-mode)
 (auto-save-visited-mode)
 (avy-linum-mode)
 (beacon-mode)
 (buffer-face-mode)
 (buffer-read-only)
 (caps-lock-mode)
 (centaur-tabs-local-mode)
 (centaur-tabs-mode)
 (cider--debug-mode)
 (cider-auto-test-mode)
 (cider-enlighten-mode)
 (cider-mode)
 (cider-popup-buffer-mode)
 (cl-old-struct-compat-mode)
 (company-search-mode)
 (compilation-minor-mode)
 (compilation-shell-minor-mode)
 (completion-in-region-mode)
 (dash-fontify-mode)
 (defining-kbd-macro)
 (delete-selection-mode)
 (desktop-save-mode)
 (diff-auto-refine-mode)
 (diff-minor-mode)
 (dired-hide-details-mode)
 (dired-isearch-filenames-mode)
 (dired-omit-mode)
 (doc-view-minor-mode)
 (doc-view-presentation-mode)
 (edebug-mode)
 (edebug-x-mode)
 (electric-layout-mode)
 (electric-quote-mode)
 (enriched-mode)
 (erc-irccontrols-mode)
 (erc-keep-place-mode)
 (erc-move-to-prompt-mode)
 (erc-noncommands-mode)
 (erc-readonly-mode)
 (erc-scrolltobottom-mode)
 (erc-smiley-mode)
 (erc-unmorse-mode)
 (evil-cleverparens-mode)
 (evil-collection-magit-toggle-text-minor-mode)
 (flyspell-mode)
 (geiser-autodoc-mode)
 (geiser-mode)
 (geiser-smart-tab-mode)
 (general-override-local-mode)
 (git-commit-mode)
 (git-timemachine-mode)
 (global-dash-fontify-mode)
 (global-hl-line-mode)
 (global-linum-mode)
 (global-prettify-symbols-mode)
 (global-reveal-mode)
 (global-semantic-highlight-edits-mode)
 (global-semantic-highlight-func-mode)
 (global-semantic-show-parser-state-mode)
 (global-semantic-show-unmatched-syntax-mode)
 (global-semantic-stickyfunc-mode)
 (global-whitespace-newline-mode)
 (gnus-dead-summary-mode)
 (gnus-undo-mode)
 (hl-line-mode)
 (horizontal-scroll-bar-mode)
 (ibuffer-auto-mode)
 (image-minor-mode)
 (isearch-mode)
 (ispell-minor-mode)
 (ivy-rich-project-root-cache-mode)
 (jedi-mode)
 (jit-lock-debug-mode)
 (line-number-mode)
 (linum-mode)
 (lispy-goto-mode)
 (lispy-mode)
 (lispy-other-mode)
 (mac-auto-ascii-mode)
 (mac-auto-operator-composition-mode)
 (mac-font-panel-mode)
 (macrostep-mode)
 (magit-blame-mode)
 (magit-blame-read-only-mode)
 (magit-blob-mode)
 (magit-wip-after-apply-mode)
 (magit-wip-after-save-local-mode)
 (magit-wip-after-save-mode)
 (magit-wip-before-change-mode)
 (magit-wip-initial-backup-mode)
 (magit-wip-mode)
 (mail-abbrevs-mode)
 (mml-mode)
 (mouse-wheel-mode)
 (next-error-follow-minor-mode)
 (org-cdlatex-mode)
 (org-indent-mode)
 (org-list-checkbox-radio-mode)
 (org-src-mode)
 (org-table-follow-field-mode)
 (org-table-header-line-mode)
 (orgtbl-mode)
 (outline-minor-mode)
 (overwrite-mode)
 (paragraph-indent-minor-mode)
 (paredit-mode)
 (prettify-symbols-mode)
 (racket-smart-open-bracket-mode)
 (rectangle-mark-mode)
 (reveal-mode)
 (rigpa-char-mode)
 (rigpa-line-mode)
 (rigpa-word-mode)
 (semantic-highlight-edits-mode)
 (semantic-highlight-func-mode)
 (semantic-mode)
 (semantic-show-parser-state-mode)
 (semantic-show-unmatched-syntax-mode)
 (semantic-stickyfunc-mode)
 (server-mode)
 (sh-electric-here-document-mode)
 (shell-command-with-editor-mode)
 (show-smartparens-global-mode)
 (show-smartparens-mode)
 (size-indication-mode)
 (slime-autodoc-mode)
 (slime-edit-value-mode)
 (slime-editing-mode)
 (slime-fuzzy-target-buffer-completions-mode)
 (slime-macroexpansion-minor-mode)
 (slime-mode)
 (slime-popup-buffer-mode)
 (slime-repl-map-mode)
 (slime-repl-read-mode)
 (slime-trace-dialog-autofollow-mode)
 (slime-trace-dialog-hide-details-mode)
 (slime-trace-dialog-minor-mode)
 (smartparens-global-mode)
 (smartparens-global-strict-mode)
 (smartparens-mode)
 (smartparens-strict-mode)
 (smerge-mode)
 (sr-modeline)
 (sr-popviewer-mode)
 (sr-tabs-mode)
 (sr-term-char-minor-mode)
 (sr-term-line-minor-mode)
 (straight-live-modifications-mode)
 (straight-symlink-emulation-mode)
 (symex-editing-mode)
 (tab-bar-history-mode)
 (tab-bar-mode)
 (tar-subfile-mode)
 (temp-buffer-resize-mode)
 (text-scale-mode)
 (tool-bar-mode)
 (transient-resume-mode)
 (tree-sitter-hl-mode)
 (tree-sitter-mode)
 (undo-tree-visualizer-selection-mode)
 (unify-8859-on-decoding-mode)
 (unify-8859-on-encoding-mode)
 (url-handler-mode)
 (use-hard-newlines)
 (vc-parent-buffer)
 (view-mode)
 (visible-mode)
 (which-function-mode)
 (whitespace-mode)
 (whitespace-newline-mode)
 (window-divider-mode)
 (with-editor-mode)
 (xref-etags-mode))
</pre>
</details>

[greghendershott]

I can reproduce this. Even more simply (just to get the racket-send-region stuff out of the picture):

#lang rhombus

1. racket-run.

2. At REPL prompt, type 1, press C-j to enter a newline without submitting, and hit RET. That prints 1.

3. At REPL prompt, enter 2. It abends ("Process Racket REPL </var/tmp/> connection broken by remote peer").

---

This seems to be because at step 1, current-read-interaction returns syntax for 1, but at step 2, it returns eof.

Upon eof, the Racket Mode REPL exits. (This closes the TCP connection for the REPL. The back end is still running.)

---

If I try type 1, press C-j, type 2, press C-j, and RET -- i.e. the input string "1\n2\n\n" -- then the next REPL interaction is OK (no eof).

So it seems like the rhombus reader is handling "1\n\n" as two reads, a 1 and and eof? (Which the racket reader does not do. It would read 1, and read nothing for the \n\n, skipping them and blocking until something following could be read.)

This seems like a bug or quirk in the rhombus reader? Maybe in command-line racket it is hidden or N/A due to the use (by default) of readline and/or expeditor??

[greghendershott]

This reminds me of #305 about #lang datalog, where EOF seemed to be used to mean "expression delimiter" as well as "end of file/input". No one I asked was able to explain why, or how this was supposed to work. For awhile I attempted to support it, but had to stop when adding support for multiple REPLs, whose I/O goes over TCP (and anyway no one seemed to be using datalog with Racket Mode).

If current-read-interaction can return non-syntax values like something satisfying eof-object?, it seems like it could return a distinct new value expression-delimiter? -- the expected handling of which is documented? At least that's my initial idea about this, but I think I probably don't fully understand what problem was trying to be solved by overloading EOF to mean other things, too.

[greghendershott]

Oh now I remember, there was more discussion here.

I'm not sure this is really the same issue but it seems "adjacent"??

[countvajhula]

Thank you for looking into this so promptly. Using a different value to avoid conflating true EOF from "expression delimiter" sounds like a good idea. Independently, I verified that this in fact only happens with Rhombus and not with a Racket source buffer, on the same 8.5 REPL (and the new title summarizes the issue well!).

[greghendershott]

It looks like this will be resolved by changing the rhombus shrubbery lexer to stop doing this: https://racket.discourse.group/t/current-read-interaction-returning-eof/1032

However I'll leave this issue open until that happens and we can confirm all is well.

(I'm still not sure what to do about datalog. If datalog could be modified to return a distinct value meaning "Do X", I'd be happy to modify Racket Mode to handle that value and do X. Meanwhile if it's not really important to anyone, I'm OK not doing anything.)

[greghendershott]

I noticed https://github.com/racket/rhombus-prototype/commit/3a293448e218b8031506411816b3d3f675aa4ecf from earlier today.

I did raco pkg update rhombus-prototype.

Now it behaves as expected.

I'm closing this, but if you do not see the same, or have another related problem, of course feel free to re-open this.

[countvajhula]

Excellent, thank you. I will reopen if I encounter it again 👍