Tree-edit POC for one transformation

[countvajhula]

Play around with tree-edit and try to use it in the implementation of a single transformation like symex-delete. Depending on what we learn from this initial POC, it will help us plan for when an integration would make sense (for instance, would it be best (a) after releasing the current work, or (b) as part of the current work, if it turns out that it will help us avoid writing hacks for corner cases and other ad hoc code), and also what such an integration might look like (e.g. are any changes required in either symex or tree-edit that would aid a seamless integration?)

[countvajhula]

FYI @ethan-leba

[ethan-leba]

Hey! Cool to see the progress that’s being made on Symex. As for changes that would be required, my thought was that tree-edit will require custom grammars for future versions that have been modified to better work for structural editing due to the issues listed here: https://github.com/tree-sitter/tree-sitter/issues/1558

I’ve also been working on porting tree-edit to emacs 29 tree sitter, is that something that’s in the works for symex?

Finally tree-edit is not completely functional yet and I likely won’t have significant time to work on it til April. I saw that there was a date in march selected as a launch date for symex.

[countvajhula]

Thanks for weighing in! And there's no rush. The March timeline is to polish up the existing tree-sitter functionality so that it has feature parity with the functionality for Lisp, as much as is practical. In the unlikely event tree-edit can shorten that timeline, I'll look at an integration in the coming weeks, but most likely, this is going to be something we'd explore after 2.0 🙂

Re: Emacs 29 tree-sitter, that's great! @polaris64 was working on a prototype for that some time ago but I haven't had a chance to try it out. At the moment, Symex relies on a small number of tree-sitter primitives, and I think that might even shrink over time in favor of relying on the DSL layer to implement most features (e.g. emit/capture ("barf"/"slurp") would be implemented once at the DSL layer instead of independently for Lisp (paredit) and tree-sitter). From a purely structural perspective, it may be that the main tree-sitter/Lisp primitive we'd need besides motions is deletion (and maybe some form of insertion..). Using the tree-sitter primitives directly for this I think would not be grammar-aware, and would allow the tree to be in invalid states, so this could be one opportunity for integration. Beyond this, if there are other tree-edit features that leverage the grammar, I assume those would not interfere with Symex functionality and would need only minimal integration or none at all. Now, I say all this without actually having much experience with tree-edit and it will still be some time before I can explore these things, so please consider these thoughts as very provisional. Any other thoughts are welcome!

[ethan-leba]

Cool! Sounds good, hoping we can slowly work towards an integration then :)

As for primitives, tree-edit has quite a few more than you've listed: deletion, insertion, raise/wrap, slurp/barf of the top of my head. It's possible that they may not be needed though? I'd be glad to hear your thoughts once you get a chance to check out tree-edit, thought it may be for the best to delay that until both our projects are on emacs29 tree sitter.

I will let you know as the project progresses!

[countvajhula]

Yeah, in Symex the DSL, emit-forward (barf) could be expressed as:

1. enter the expression
2. move all the way to the right
3. delete
4. exit the expression
5. paste-after

The code could look something like this:

(symex-traversal
  (do (move up)
      (keep (move forward))
      (delete 1)
      (move down)
      (paste after)))

(Note this is pseudocode as the DSL in its current form doesn't support transformations like delete, and some of these forms above currently have different, arguably more idiosyncratic names!)

So here the primitives needed are only the motions, delete and paste. I think most features could probably be implemented with just these. Naively, this code would always work, and it might leave the tree in a weird state if deletion was not allowed for some reason (or maybe if paste led to a weird state). By using the rich primitives provided by tree-edit, I think such cases would fail and preserve the integrity of the tree allowing for a more robust UX. Of course there are probably many other features in tree-edit and I suspect they would be compatible with Symex features since they have a similar UI.

In any case, yes, I'll keep you updated as well!

[ethan-leba]

Ah, makes sense! That's essentially the same way tree-edit's slurp works as well. I've just pushed a big update to tree-edit for emacs29 treesit and a ton of misc changes, so I would be very happy to get any feedback now once you get a chance to test it out!

[countvajhula]

That's great news, congrats on pushing that work through! I will aim to look at this soon, closer to the 2.0 release or maybe right afterwards. That's still planned for "Spring" but obviously the March timeline was ambitious.