Not taking SET-MACRO-CHARACTER into account?

[kpoeck]

i have trouble loading maxima with the cst version of clasp, while it loads fine with the ast-version. The cst-version of clasp uses eclector.concrete-syntax-tree:cst-read to read lisp-sources.

The problem seems to be the following (tested with sbcl and eclector from git)

Define a marco character "_" (the silly pprints are just to test)

(ql:quickload :eclector)
(ql:quickload "eclector-concrete-syntax-tree")
(set-macro-character #\_ (lambda (stream char)
                             (pprint `(char = ,char))
                             (let ((was (peek-char nil stream nil nil t)))
                               (pprint `(peek = ,was))
                   (case was
                 (#\" (values))
                 (#\N (read-char stream t nil t) (values))
                 (otherwise '_))))
                       t)

Use that

* (read-from-string "_N\"The message-lookup\"")
(CHAR = #\_)
(PEEK = #\N)
"The message-lookup"
22

but than in eclector:

* (eclector.reader:read-from-string "_N\"The message-lookup\"")
_N
2
* (with-input-from-string (stream "_N\"The message-lookup\"")
  (eclector.concrete-syntax-tree:cst-read stream))
#<CONCRETE-SYNTAX-TREE:ATOM-CST raw: _N {10046D7143}>
NIL

Am I missing something or are user-defined macro-characters not used?

[scymtym]

I'm not sure how Clasp uses Eclector, but considering your example, the problem is that cl:set-macro-character affects the "host" readtable. Eclector has is own readtable class and thus *readtable* variable (the "host" could restrict cl:*readtable* to values of its own readtable type). Its name is eclector.readtable:*readtable*.

Therefore, when running inside e.g. SBCL, something like the following is needed

(eclector.readtable:set-macro-character
 eclector.readtable:*readtable*
 #\_ (lambda (stream char)
       (pprint `(char = ,char))
       (let ((was (peek-char nil stream nil nil t)))
         (pprint `(peek = ,was))
         (case was
           (#\" (values))
           (#\N (read-char stream t nil t) (values))
           (otherwise '_))))
 t)

(with-input-from-string (stream "_N\"The message-lookup\"")
  (eclector.concrete-syntax-tree:cst-read stream))
(CHAR = #\_)
(PEEK = #\N)
#<CONCRETE-SYNTAX-TREE:ATOM-CST raw: "The message-lookup" {100BA4C2F3}>
NIL

If Clasp's cl:set-macro-character modifies the value of cl:*readtable*, Eclector won't pick those changes up.

[kpoeck]

As far as I can see, clasp simply uses the following integration pattern:

• eclector.reader:client is set up
• eclector.concrete-syntax-tree:cst-read is used, when the compiler reads lisp-source
• eclector.parse-result:source-position is customized
• eclector.reader:find-character is customized

Looking at your answer, that won't work - as I have seen it compiling maxima. I assume the following have to be added:

• loading files needs to use eclector as well
• all operations on readtables need to
  • use the eclector readtable directly or
  • need to inform eclector about changes in the readtable (and how this could work with (copy-readtable ..) is not clear to me
• cl:read-from-string needs to use eclector.reader:read-from-string
• cl:read-preserving-whitespace needs to use eclector.reader:read-preserving-whitespace
• cl-read needs to call eclector.reader:read and in an earlier conversation you told me to consider
• ECLECTOR.READER:MAKE-STRUCTURE-INSTANCE and
• ECLECTOR.READER:WRAP-IN-*

This list might be incomplete or partially wrong. Is there already a documentation for such an integration?

[scymtym]

loading files needs to use eclector as well

I'm not sure what this means. Doesn't Clasp use Eclector as its reader and thus for loading files?

• cl:read-from-string needs to use eclector.reader:read-from-string
• cl:read-preserving-whitespace needs to use eclector.reader:read-preserving-whitespace

This is an interesting point. I haven't thought about whether you can do

(let ((eclector.reader:*client* <a-cst-client>))
  (eclector.reader:read-from-string …))

i.e. using a CST client with the ordinary read functions. I think it should work, but I haven't tested it.

Doing it this way may do lots of unnecessary work constructing parse results that are not going to be used, though. So maybe having separate classes clasp-client (for shared behavior), clasp-raw-client and clasp-cst-client is the best approach.

This list might be incomplete or partially wrong. Is there already a documentation for such an integration?

If you are asking for a "How to use Eclector as the reader of a Common Lisp implementation" section - No that does not exist yet, but it seems like a good idea.

[Bike]

At the moment, Clasp only uses Eclector to read files it is compiling. Everything else (explicit calls to read, load, etc.) uses Clasp's own reader.