search

drym-org / symex.el

An intuitive way to edit Lisp symbolic expressions ("symexes") structurally in Emacs
Other
271 stars 22 forks source link

Symex 2.0 Release Prep #71

Open countvajhula opened 1 year ago

countvajhula commented 1 year ago

Summary of Changes

All the changes needed for a 2.0 release, including polishing the tree-sitter functionality, improvements and bug fixes.

Remaining Work

See the ✨ Kanban Board ✨

WIP

Already Done

(reverse-chronological, most recently completed at the top)

Public Domain Dedication

(Why: The freely released, copyright-free work in this repository represents an investment in a better way of doing things called attribution-based economics. Attribution-based economics is based on the simple idea that we gain more by giving more, not by holding on to things that, truly, we could only create because we, in our turn, received from others. As it turns out, an economic system based on attribution -- where those who give more are more empowered -- is significantly more efficient than capitalism while also being stable and fair (unlike capitalism, on both counts), giving it transformative power to elevate the human condition and address the problems that face us today along with a host of others that have been intractable since the beginning. You can help make this a reality by releasing your work in the same way -- freely into the public domain in the simple hope of providing value. Learn more about attribution-based economics at drym.org, tell your friends, do your part.)

countvajhula commented 1 year ago

Hey all, hope everyone's doing great. I'm getting the ball rolling on this effort for anyone who's around to work on it 😊 I'll aim to document outstanding items more thoroughly in the coming days, but I think the Kanban board from last time should still roughly be current if you want to take a crack at anything in particular. I think we made an effort last time to keep the tasks pretty bite-sized. I'll review them more closely and update them as needed soon. Get pumped for Symex 2.0! 😉 👍

@markgdawson @anonimitoraf @pepperblue @doyougnu @dcostaras @devcarbon-com

countvajhula commented 1 year ago

Thanks for reviewing the changes already in the branch @j-shilling ! Just to clarify, this branch is going to serve as the integration branch for pre-2.0 changes (including the WIP PRs #69 and #70 which will be merged here as opposed to master). This is partially as a way to organize for collaboration, and partially so that we can move faster as we only need to worry about testing everything once, at the end, rather than in depth for every PR.

countvajhula commented 1 year ago

I've migrated and updated the Kanban board (added in the PR description) and converted all tickets into actual issues on the repo, and also added any relevant user bug reports to the board. I think the tasks are all ready to be worked on. For the numerous tree-sitter features, there are no descriptions in the issues as they should function analogously to the Lisp features. Happy to answer any questions or help in any way! I'm not a huge chat user nowadays but I'll aim to hang out in #emacs IRC as another communication option in case that's easier for anyone. If you are interested in contributing, feel free to shoot me a message there!

countvajhula commented 1 year ago

For anyone who wants to play with the latest and greatest in this branch (there are lots of improvements already, and of course more exciting work still planned 😸 ), you could use config resembling this:

(use-package symex
  :straight
  (symex
   :type git
   :host github
   :repo "drym-org/symex.el")
   :branch "2.0-integration"
   :config (symex-initialize))

If you're interested in working on anything in particular, comment here (or find me on IRC if I happen to be there) and I'll assign the ticket to you. Help would especially be appreciated on:

devcarbon-com commented 1 year ago

Is the plan to replace tsc with the built-in treesit library?

countvajhula commented 1 year ago

@devcarbon-com That is the eventual goal, yes. It isn't currently planned to be changed as part of the 2.0 release, but I know @polaris64 has some WIP on this front so if he champions it or if it turns out to be a simple change we might include it.

devcarbon-com commented 1 year ago

Okay, sounds good. A precursory look seems like it will be pretty straight forward, except for maybe tsc-changed-ranges, I haven't seen a equivalent for that yet.

countvajhula commented 1 year ago

Hey folks! In case you've been using this branch, there were some bugs that had been introduced a little while back (e.g. in emit and capture) that were recently fixed in #123 , and there were a number of performance improvements in #124 and #125 , so you should update to the latest to get these 🙂

One of the performance improvements is in the "leap branch" feature, which now executes in O(n) where formerly it was O(nlogn). This should make it pretty seamless to use and in my experience it doesn't cause noticeable delays in most cases as it used to formerly. Some other performance improvements were in (1) making traversal execution tail-recursive, which allows it to be more memory efficient (since the in-progress results are passed into recursive calls instead of retained in the calling scope to be combined with the result after recursion). And (2) the "recursive" features like recursive tidy/indent (M-tab), recursive eval (M-e) were formerly causing some delays in execution as well, but now they are pretty zippy! In fact, when I tried recursive tidy on the master branch it actually caused Emacs to crash 😅 . Not sure if that's just me or if it's actually doing that at the moment - that would be bad, lol. In any case, these issues are fixed in this branch as it is more memory-efficient all around. The DSL is also more expressive now, with the ability to express conditional recursion via a new loop form, and arbitrary side effects via effect, and that should allow us to implement features (including existing features) more conveniently going forward.

There are a couple of introduced bugs, however -- (1) counts/quantifiers don't work for deletion at the moment, and (2) "soar" across trees (M-} M-{) operates like "leap" (} {) within a tree, instead of always crossing trees. I'm working on it and will hopefully have fixes soon, though I need to take a short break to finish some other stuff.

Enjoy, and please let me know if you run into any issues while using this branch.

devcarbon-com commented 1 year ago

Hurray! :tada:

Also, one of the the "bugs" sounds like a feature I want - a leap-or-soar command :D

countvajhula commented 1 year ago

Good to know! One of the reasons leap doesn't already soar is because accidentally crossing trees formerly had a high cost. Since the cost isn't as high now, we could potentially make leap-or-soar the default behavior, with soar being an explicit "leave this tree."

dcostaras commented 1 year ago

Hi @countvajhula :)

Apologies for being so quiet, it hasn't been because I'm not excited by Symex and Rigpa, work has just been nuts for the last 2 years.

For better or worse that has now come to an abrupt end and I'd love to be at least a bit more involved in the projects. I'm going to start by just catching up with the developments of the last year and a half.

Cheers, d

countvajhula commented 1 year ago

Hi @dcostaras . Congrats! This is the first step on the road to liberation from the reductive capitalist imperatives that imprison us all! 😆

Less flippantly, I'm glad to hear that. Welcome back. I'm currently taking a short break from Symex to focus on Jewel, but I'll be returning to work on 2.0 soon.

Re: what you could do, I think directly helping on 2.0 release work (e.g. tree-sitter, package decomposition, DSL, refactoring, etc.) would be tricky to coordinate in the immediate timeframe, but here are some broad things on my mind that are more decoupled and could be good to work on:

I realize that's a lot of stuff to mull over but in case you can help with any of it, that would be helpful and appreciated. If you decide to work on any component, please do reach out to anyone else who may be on the relevant threads in case they are interested in collaborating or if they have any thoughts or WIP on it. Let me know if I can help clarify anything in particular!

dcostaras commented 1 year ago

Hi @countvajhula haha, don't get me started on capitalism, the ultimate local maximum!

Thanks for the detailed response, in reading the mentioned threads and other issues your thorough communication is beneficial :)

Due to a rising sense that my Doom Emacs config and Evil/Symex hacks (which would be almost entirely solved by using Rigpa) are starting to grow a life of their own (maybe time to declare emacs config bankruptcy again!) and some issues with RSI, I've been thinking about how to do things differently. I've never managed to get Rigpa setup properly on my config; so, I think, that is where I'm going to start. But also I've been looking at other modal paradigms such as Kakoune and Helix and other Emacs model systems like Meow and Boon (@devcarbon-com, it looks like you're also currently looking at Meow specifically). This all makes me quite interested in the work towards a cleaner modal system for use in Rigpa and Symex. I know that's not a good first goal as I don't have the required experience with Evil, Symex and Rigpa and it's not an immediate priority for Symex/Rigpa anyway but that's what's tickling my pure aspirational and intellectual interest.

Once I've got Rigpa and Symex WIP 2.0 working I'll have a look at the above smaller, more immediate tasks.

dcostaras commented 1 year ago

BTW, is general project communication happening on GH issues/PRs and the Symex 2.0 board?

countvajhula commented 1 year ago

Yes, general project communication is happening on GitHub Issues/PRs and there are no other channels in use at the moment.

If you have any difficulties setting up Rigpa or if the instructions are inaccurate / non-ideal / don't work out of the box for any reason, please report that as it's a bug 🙂

RomeoV commented 1 year ago

Excited to see this come to life soon! For now, what I really only want is some vim-like navigation of the syntax tree for non-lisp languages with emacs-29 treesit, e.g. for org mode. For example, moving in around the headings in org document using hjkl. Is this already viable if I have an org-treesit grammar installed?

countvajhula commented 1 year ago

Hi @RomeoV ! Being able to navigate org documents with hjkl would be great, and I'm curious what would happen if you tried that right now with org-treesit together with Symex. Assuming tree-sitter-mode is true, it should trigger structural navigations in the alpha support Symex already has for tree-sitter in the main branch, and it's possible it does work already (I've tried this with navigating JSON documents in the past and that has worked fine).

Now re: the 2.0 effort, it's high time for a status update so here it is:

I believe the 2.0 work is reasonably close, but there's only so much time in the day and lately I've been focusing on Attribution Based Economics (which Symex is one of the pilot projects for! I'm very excited about the possibilities that ABE offers for open source economic viability as there have been some very interesting breakthroughs lately that I believe resolve some questions I myself have felt were unanswered by ABE thus far), and recently gave a talk at the Foresight Institute on that subject. I will be giving another talk related to ABE at RacketCon in October, so I don't think I will have a lot of time for a deep push on 2.0, but it's silly for so many improvements to be sitting in a branch not being used.

So here's what I'm thinking: I can aim to do a triage of the 2.0 work in the near future and retain anything that is a clear improvement over master, while being more conservative with changes that aren't quite there yet. The goal would be to merge as much of this into main as possible so that further work can continue on the main branch.

What isn't done yet:

  1. The initial goal was to have the Symex DSL be a level of abstraction above the primitive tree operations (i.e. tree-sitter and native ELisp), so that most/all structural features would be implemented at a high level of abstraction in Symex, and this would allow the primitives implemented by tree-sitter and Lisp to be relatively small instead of having large sets of primitive operations. We aren't especially close on this front since drawing the new abstraction boundaries will take some time, as Tree-sitter in some ways accomplishes what was initially unique about Symex -- that is, being able to assume an explicit tree representation in implementations of features. I still think it would be good to aim for this since Symex is a higher level language than tree-sitter, and thus easier even for casual users to add nontrivial features in. But as I said, we aren't especially close yet on this front and it will take more work than I can do in the immediate future, and having to use a large set of primitives to provide various features may be OK for now.

  2. Tree-sitter UX polish. There are still a number of edge cases in the UX with respect to tree-sitter languages, and it would be good to tackle these one at a time as they come up.

  3. Using Emacs's native Tree Sitter either directly or via tree-edit. This wasn't explicitly a goal with this release but it would be a nice one to tackle soon as many folks have brought it up.

  4. Package decomposition. Symex is a big package. We want to decompose it into several small packages (e.g. DSL, interface, runtime, etc. in separate packages). We just might be able to do this as part of this release.

  5. One big improvement I wanted to make was actually in Rigpa, to use the dynaring package in ways that would make the UX with Symex and Normal state very natural. But it does not seem likely I'll be able to get to it this time around.

  6. Reducing reliance on Evil. This one's on @devcarbon-com 🙂

On next steps, I will aim to triage this branch and get it into a reasonably mergeworthy form. At that stage, we could use help with testing on both Lisp and especially Tree Sitter. I will post about that when we're there.

Thanks again for your interest and support.

RomeoV commented 1 year ago

Thanks for the detailed answer. Just one thought from my side: I do think with the inclusion of treesit in emacs, many people are looking to switch away from having both tree-sitter-mode and treesit installed. For me personally, any package focussing on tree-sitter-mode is a non-starter, as it's most likely not the future.

I think a triage for 2.0 would be useful indeed, there can always be a 3.0 release ;) Good luck also with your other work!

countvajhula commented 3 months ago

Heads up: I will be rebasing this branch sometime today to bring it up to date with master.

countvajhula commented 3 months ago

This branch has been rebased! If you happened to have work in progress that was based on this branch, you could use these instructions to bring it up to date and reflect the updated base branch correctly.