lispy-tab breaks symbols ending with '

[yuhan0]

In symbolic or mathematical code the single quote ' character is commonly used at the end of a symbol as a substitute for the prime mark ′. This is of course a valid lisp identifier, but lispy-tab interprets it as a quote belonging to the next symbol and shifts it over to the right, completely changing its meaning:

|(defn newton-second-law [x mass step]
  (let [x' (diff x step)
        x'' (diff x' step)]
    (fn [t]
      (* mass (x'' t) ))
    ))

;; after lispy-tab

(defn newton-second-law [x mass step]
  (let [x '(diff x step)
        x ''(diff x 'step)]
    (fn [t]
      (* mass (x ''t)))))

[yuhan0]

Update: when a symbol ending with ' occurs before a closing delimiter anywhere within a form, the entire form is replaced with nil. It doesn't throw any errors making it hard to pinpoint the root cause.

;; before
(foo x')

;; after lispy-tab
nil

;; before
(define (test-inverse A inv)
  (let ((a (pick-random A))
        (a' (inv a)))
    (check-equal? (inv a') a)))

;; after lispy-tab
nil

[abo-abo]

This is of course a valid lisp identifier

x' is not a valid lisp identifier in:

• Elisp
• Common Lisp (checked with SBCL)
• Scheme (checked with Racket)

It's a valid identifier only in Clojure.

I suggest to rewrite your code like this:

(defn newton-second-law [x mass step]
  (let [xp (diff x step)
        xpp (diff xp step)]
    (fn [t]
      (* mass (xpp t)))))

If you want to add a patch to support identifiers with ' in lispy--read, go ahead.

[yuhan0]

Thanks for the reply! I tested the following in both Elisp (in Emacs 26.1) and DrRacket, it seems like x' is in fact a valid identifier, except for the fact that the lisp reader can't handle a quote directly before a closing paren:

(let ((x' 10))
  (+ x' 5))
;; => 15

(let ((x' 10))
  (+ 5 x'))
;; Invalid read syntax

I'll look into lispy--read when I have the time, thanks 🙂

[yuhan0]

PS. I did rewrite some of my (Clojure) code with such identifiers after discovering this issue, the concern I had was that a "harmless" formatting operation could result in syntactically valid code which would still compile and run but with different semantics, and the change is hard to make out visually.

As a contrived example:

;; Before:
(defn fail-safe [x]
  (let [x' (fn-which-returns-nil)]
    (when (and x' x)
      (launch-missiles!))))

;; After (spot the difference): 
(defn fail-safe [x]
  (let [x '(fn-which-returns-nil)]
    (when (and x 'x)
      (launch-missiles!))))

[abo-abo]

The issue stems from the fact that the Elisp reader is used to read LISP code written in other dialects.

The advantage of this approach is that we don't need to interact with a foreign LISP REPL for i. The disadvantage is that we're using semi-dumb regexes to make the foreign LISP code readable by Elisp read.

Maybe you'd like to experiment with the alternative approach: Emacs sends the defn string to Clojure saying "pretty-print it", and Clojure uses its own reader or libs to pretty print the code and return it as a string.

[yuhan0]

Another instance of this bug with the same root cause: using lispy-tab on symbols with the ? character sometimes replaces it with \?. These are the same to the elisp reader but not to Clojure.

;; original
(s/? ::spec)
;; after lispy-tab
(s/\? ::spec)  ;; error: invalid token

;; elisp
(eq 'a1? 'a1\?)
;; => t

;; clojure
(= 'a1? 'a1\?)  ;; actually parsed as 3 separate tokens a1?, a1 and the character \?
;; => false

This only seems to happen when ? is preceded by non-alpha characters, like numbers or \ or :

The latter 2 are especially problematic for existing libraries out there like clojure.spec.alpha which use vars and keywords starting with ?, see the example above.

[yuhan0]

Okay, I narrowed it down to https://github.com/abo-abo/lispy/blob/master/lispy.el#L8058 in the function lispy--insert.

Replacing that regex with "\\(?:\\s_\\|\\sw\\)\\(\\\\\\?\\)" appears to solve the latter issue.

[abo-abo]

Replacing that regex with "\\(?:\\s_\\|\\sw\\)\\(\\\\\\?\\)" appears to solve the latter issue.

Thanks, applied.