Comments on Atom escaping

[ejgallego]

Dear sexplib developers,

We have been using sexplib to talk to emacs, however, we have found sexplib's escaping to be a bit weak with regards to Emacs parser. In particular, Atom "," or Atom "`" don't seem to be escaped, confusing the emacs (and other libraries like node sexp, see ejgallego/coq-serapi#22 for more details).

We have fixed this by implementing our own sexp escaping and printing, but we wonder if some action should be taken in sepxlib. I can think of two options:

• extending sexplib to allow the user to provide a custom escaping routine (this seems like a bad idea).
• make escaping stronger in sexplib so that leading commas, backquotes, and #' are escaped.

What do you think? We'd be happy to provide a patch (and do the paperwork etc...) if you think it would make sense.

Best regards, Emilio

[ghost]

I imagine that if you are talking to a third-party application that has its own rules for interpreting s-expression like syntax, it's better to write your own printer so that you are sure it is correctly interpreted on the other side.

Do you know if there is some kind of standard for s-expression like syntax?

[aalekseyev]

Personally I think it's reasonable if the quoting function escapes more than it absolutely needs to, if it helps clarity or compatibility. For example it's nice that we escape the non-ascii bytes. The change to always quote punctuation seems good to me in principle.

However, Sexp.to_string has been stable for so long and used a lot, so I'm hesitating to actually change it merely for this, in case some projects assume stability. Maybe we should have a compromise where Sexp.to_string_hum quotes punctuation, but Sexp.to_string does not?

[ejgallego]

Thanks for having a look @aalekseyev , indeed it seems a bad idea touching that function, a new to_string_quot [with even an optional parameter] could be the way to go.

I am very far from being knowledgeable in lisp myself, I'll ping @cpitclaudel who I'm sure could provide better feedback here [other than what we have in the issue]

[cpitclaudel]

In the long run, I think SerAPI should move to JSON — possibly just encoding s-exps in JSON (basically quote everything and use [ instead of (). This is because few languages have good sexp-parsing code; for example Alectryon (written in Python) spends a good 30-50% of it's time parsing and unparsing sexps, because there's no native Python library for Sexp parsing. And now that Emacs has native JSON, it would probably not be any slower if SerAPI produced JSON.

[ejgallego]

Ok, so maybe we should close this issue and the one in SerAPI?

I'm myself not an expert on the quoting issue.

Note that sexp was chosen for SerAPI because I understand is the most mature serialization backend, and I am reluctant to remove it for that same reason. SerAPI already supports direct serialization to both native Python and JSON objects, tho some issues do remain. But indeed having three different codebases doing essentially the same in [@@deriving sexp,yojson,python] makes for quite a bit of thinking....

[cpitclaudel]

SerAPI already supports direct serialization to both native Python and JSON objects, tho some issues do remain

!! I didn't realize!

[aalekseyev]

I'm closing this issue. It's clearly not the right way to go for compatible {de,}serialization, and that seems to be the main reason people are interested in this.

[ejgallego]

Thanks for the feedback and for triaging this @aalekseyev