search

s-expressionists / Eclector

A portable Common Lisp reader that is highly customizable, can recover from errors and can return concrete syntax trees
https://s-expressionists.github.io/Eclector/
BSD 2-Clause "Simplified" License
109 stars 9 forks source link

read-from-string's source don't take into account `:start` #75

Open fstamour opened 2 years ago

fstamour commented 2 years ago

I'm not sure if it's on purpose or not, but when using (read-from-string ... :start n) with a custom client, the positions returned are relative to n instead of being relative to the start of the string.

So (read-from-string "0123x" :start 4) would return a node corresponding to x but with the source (0 . 1) where I expected (4 . 5).

For now, I'll just use the workaround of storing start in the client and add that to the source in the right methods...

scymtym commented 2 years ago

Thank you for the report. Unfortunately, the issue is caused by implementation specific behavior. Eclector uses

(with-input-from-string (stream string :start start :end end :index index)
  (read-aux client stream eof-error-p eof-value preserve-whitespace))

to read from strings since the reader is specified and implemented in terms of streams. Source positions, by default, use file-position for which no behavior is specified on string streams. And indeed the behaviors of implementations differ:

;;; SBCL
(with-input-from-string (stream "foobar" :start 5) (file-position stream)) => 0
;;; ECL
(with-input-from-string (stream "foobar" :start 5) (file-position stream)) => 5

One solution would be wrapping the string stream in some gray stream and do the position tracking that way, but the performance implications seem prohibitive.

Another solution would be storing a source position offset in the client object and applying that, for example, in an :around method on source-position. But then there would have be a way to determine the implementation specific offset which should be applied (In addition to the cost in terms of performance and complexity).

In summary, I don't see a good way forward. Any suggestions?

fstamour commented 2 years ago

use file-position for which no behavior is specified on string streams

Good thing I switched from reading string streams to reading string directly, because I was abusing file-position haha. I would've had a nice surprise when starting to test on different implementations...

Unfortunately, the issue is caused by implementation specific behavior.

Yes :( I saw moments after creating the issue.

Any suggestions?

I figured this behaviour should be documented at least.

Finally: my workaround worked like a charm.