Invalid read causes REPL to fail

[rvs314]

If you write #rx"\?$" (which is an invalid regex) in the REPL, you get the following result:

> #rx"\?$"

; :31:0: read-syntax: unknown escape sequence `\?` in string
; $: undefined;
;  cannot reference an identifier before its definition
;   in module: top-level

which is fine - the problem is that future inputs are broken:

> 1 ; hitting return here does not submit the completed form
2 ; hitting return again
; still waiting for more input

at this point, you can hit C-c C-c to exit the prompt altogether, which puts the REPL into a working state. I've been able to also get it to complete the form after some combination of quotes and invalid identifiers, but never into a working state.

[sorawee]

Yeah, this is unfortunate. Here's a similar problem:

> (add1 2) "abc
3
> 

Then the REPL are in the weird state that you described. If you input the same again, it reveals what's actually going on:

> (add1 2) "abc
3
> (add1 2) "abc
"abc\n(add1 2) "
; abc: undefined;
;  cannot reference an identifier before its definition
>

What happens here is that after you enter (add1 2) "abc, (add1 2) is evaluated, and "abc is incorrectly left in the buffer. That's why when we enter (add1 2) "abc again, \n(add 1 2)"abc is joined with the previous input, which evaluates to a string "abc\n(add1 2) " and an unbound id abc.

Racket Mode appears to have heuristics to prevent unmatched quotes from being submitted, but the heuristics fail here. Moreover, the fact that the heuristics sometimes work is what prevents us from getting back to the working state again (without using ctrl-c). Ideally, we can just enter " to finish the unpaired quote. But the heuristics prevent us from doing that.

[greghendershott]

Thanks for reporting. I dogfood the REPL a lot. I make plenty of mistakes. I think the reason I never noticed this is because I use paredit; as a result I rarely have unmatched delimiters like parens and quotes.

---

The Racket Mode REPL seeks to emulate DrRacket's in the sense that you may enter one or more expressions. When you RET, nothing happens until it's complete:

• If a lang supplies a drracket-submit-predicate, it uses that to decide.
• Otherwise it uses a little Emacs Lisp helper function, racket--repl-complete-sexp-p -- which is what happens in your case.

The problem seems to be that racket--repl-complete-sexp-p currently checks for at least one complete expression -- but that doesn't account for there being multiple expressions, the last of which might be incomplete, as in your example.

Something like the following seems to be a reasonable fix in my quick testing, but I'll think a bit more before committing.

modified   racket-repl.el
@@ -161,23 +161,26 @@ end of an interactive expression/statement."
       (when prefix (process-send-eof proc)))))

 (defun racket--repl-complete-sexp-p (proc)
-  "Is there at least one complete sexp at process-mark?"
+  "Is there at least one complete sexp at process-mark?
+
+When multiple sexps, all must be complete, e.g. issue #646."
   (condition-case nil
       (let* ((beg (marker-position (process-mark proc)))
              (end (save-excursion
                     (goto-char beg)
-                    ;; This will scan-error unless complete sexp, or
-                    ;; all whitespace.
-                    (forward-list 1)
+                    (while (< (point) (point-max))
+                      ;; This will scan-error unless complete sexp, or
+                      ;; all whitespace.
+                      (forward-list 1))
                     (point))))
        (not (or (equal beg end) ;nothing
                 (string-match-p ;something but all whitespace
                      (rx bos
                          (1+ (or (syntax whitespace)
                                  (syntax comment-start)
                                  (syntax comment-end)))
                          eos)
                      (buffer-substring beg end)))))
     (scan-error nil)))

[greghendershott]

TL;DR: I merged a commit for this.

I was slightly delayed because I was getting failing tests here on CI, that, while "in the same neighborhood", actually had nothing to do with my change. In fact tests passed when I ran locally.

In literally the last week, paredit made some changes, including stealing the RET key from racket-repl-submit, and the tests run here on CI use the bleeding edge version of paredit. Just resolved that (I hope) in #647.

Sometimes a coincidence is just a coincidence.

[rvs314]

I believe that the merged commit fixes @sorawee's example, but not mine - writing #rx"\?$" still ends up in the limbo-state described before (at least for me). Let me know if this can still be reproduced

[greghendershott]

Sorry!

I misunderstood @sorawee to be giving another example of the same problem, and lost track of your example.

Actually it seems these are two distinct problems.

I can still reproduce your example, in the Racket Mode REPL. :disappointed:

---

The thing is, it's even "worse" when I try using plain command-line Racket run from a plain terminal:

$ racket
Welcome to Racket v8.7.0.6 [cs].
> #rx"\?$"

  12

The first line is not even submitted as an expression; no error message, no new prompt. It's just "hung" sooner. Must C-c to proceed.

Note that this does use xrepl and expeditor enabled by default in newer Rackets. So it's default, but not exactly "plain".

For something simpler, I try:

$ racket -e '(read-eval-print-loop)'
> #rx"\?$"
stdin::1: read-syntax: unknown escape sequence `\?` in string
  context...:
   /home/greg/src/racket-lang/racket/collects/racket/repl.rkt:11:26
> $: undefined;
 cannot reference an identifier before its definition
  in module: top-level
  context...:
   body of top-level
   /home/greg/src/racket-lang/racket/collects/racket/repl.rkt:11:26
> 1
2
3

Which behaves more like Racket Mode's REPL: I get the error message, and another prompt, but thereafter it's "hung".

---

So... I'm not immediately sure what's going on here. It seems more to do with Racket's read-eval-print-loop itself, then about Racket Mode. I'm not declaring "won't-fix" or "can't-fix". I will need to think more, and maybe get some help.

[greghendershott]

OK. So, I think the most minimal form of your example is: "\?".

What seems to happen here is that read consumes the "\? from input, and the error message is given, escaping read. The final " remains in input.

You get a new prompt. read is called again; it consumes that already-waiting ", and is in the state of reading a string literal, and won't return until it sees a matching/closing ".

So in variations of your example, you can regain control by entering ". (Whether in Racket Mode or using plain racket -e '(read-eval-print-loop)').

Assuming I'm understanding that correctly, I'll try to think of what, if anything, Racket Mode's REPL can or should do differently in this regard.

[greghendershott]

DrRacket seems to discard that extra input upon an error, therefore you get an error message, a fresh prompt, and a clean slate. Which I think is what you'd want/expect, @rvs314.

I could look at doing something similar. I'm a little concerned about causing new problems. e.g. How much input to discard? What about running programs in the REPL, that themselves read input? And so on. As a result, I'm unsure about the cost:benefit or "ROI" of attempting something like this.

[greghendershott]

I haven't forgotten about this, but not confident the right thing to do.

I just pushed (to a topic branch for now -- not merged) a commit that, upon a read error, consumes input through the next newline.

I'm not sure whether this is a smart, focused mitigation, or, a short-sighted hack that will create new bugs.

[greghendershott]

To follow up: Just now I did end up merging a commit fd44a86, which directly addresses the specific issue @rvs314 you originally reported.