Get community maintainers onto mainline community repo

[pokey]

The problem

Today, none of the community maintainers are using the main branch of community. This fact means that

• we do not experience issues that may happen for new users starting out
• it is harder for users to reproduce commands they see in videos (see https://github.com/cursorless-dev/cursorless/discussions/547)
• it is hard for users to move from the relatively verbose commands in community to commands often used by more experienced voice coders (see #1221)
• it will be difficult for us to move to the future in which Talon has a package manager

The solution

The idea would be that core maintainers would be able to use the main branch of community, with their own customization repo that sits beside community to enable their own customization. Generally speaking, this will require us to make community more customizable, eg by

• Adding actions that users can override
• Moving more things to .talon-list files

These are just general principles; in practice we will need to look at the different places that our core maintainers have changed in their forks, and treat them on a case by case basis. For each change, either:

• The change is a pure improvement to community that just needs to be upstreamed, or
• The change is a personal preference that should be customizable via some mechanism in community, or
• The change is orthogonal to community, and should be able to live in a side repo that the given maintainer clones alongside community without any changes to community

How do we get there

We propose to create a git subtree of community (similar to what we do with https://github.com/talonhub/community/tree/main/apps/vscode/command_client). This subtree would contain a growing subset of community that we expect all core maintainers to leave untouched, or at least try to keep close to main and upstream changes fairly regularly. The subtree would have a well-defined api surface. Ideally it would be generic enough that @AndreasArvidsson could have this subtree cloned alongside his repo, after having removed the corresponding piece from his repo

Initially, the subtree would have a single directory from core, eg formatters or numbers. We then add more and more to it, until it grows to be most (all?) of the community repo

When Talon has a package manager, the subdirectories from this subtree could become Talon packages, and the maintainers would be able to start using them right off the bat, because they wouldn't need their own custom versions

Related

• https://github.com/talonhub/community/issues/1384

Thanks to @jaresty for pushing in this direction

[pokey]

We're planning to do a mob coding session in one of the cursorless meet-ups where we try out this idea for one small part of community, eg numbers or formatters, to see if it is promising

[jaresty]

I'm excited to see interest in trying this out! Thank you for putting in the time to experiment. This plan sounds like a really good strategy to me, and I'm excited to hear how it goes.

[pokey]

@AndreasArvidsson and I discussed formatters at length. It appears there are several PRs to get formatters to support our use cases:

• @AndreasArvidsson has improved core machinery for formatting text that is more robust eg in cases where there are non-alphanumeric symbols, some of the words are already capitalized, etc. We can lift that machinery to improve community formatter behaviour. That code can be found in the first half of his formatters.py. @AndreasArvidsson has volunteered to open a PR in that direction when he gets the chance. Note that there will be some minor changes to behaviour, but for the most part they are pure improvements, and if there are things that are more personal preferences, we can discuss that on the PR
• Migrating the formatter spoken form map to a csv or talon list. Today it is defined in https://github.com/talonhub/community/blob/027d7d0d874b9718226d8dd8bcbb96abcb4a6a1d/core/text/formatters.py#L173-L195 There are some things to think about here, because if you look at @AndreasArvidsson's formatters above, you can see that we have a similar issue as in other places where different formatters will end up in different lists. Ie some formatters are only for insertion (eg "say" as in "say hello world"), some only for reformatting (eg he has a "cap" formatter that will capitalize the text without touching anything else, unlike sentence case, which would lowercase things), and then some that are useful for both. This is a common pattern where we have two lists, and need to represent their intersection and both directions of set difference. It can make a talon-list awkward, so a csv might be the way to go
• Adding support for so-called "phrase enders", which are a set of symbols that can be used to terminate prose formatters, which otherwise need to be terminated with "over". This allows you to say eg "say hello world testing dot" instead of needing to say "say hello world testing over dot". Both @AndreasArvidsson and I have this machinery, so either of us, or someone else could upstream it. See eg my https://github.com/pokey/pokey_talon/blob/923f656680765bd8637a7a45f1a1572a76799db5/core/text/formatters.py#L365-L371 (and its use in talon file)
• Adding support for so-called "immune symbols". In community, it is possible to put any symbol in a formatter, eg "camel hello world dot whatever" to get helloWorld.whatever. @AndreasArvidsson and I both found this to result in too many false positives, so we just support it for a couple symbols like . and -. See eg https://github.com/pokey/pokey_talon/blob/923f656680765bd8637a7a45f1a1572a76799db5/core/keys/keys.py#L180-L183, and search around my repo for immune

Each of the above could be a separate PR

[pokey]

We also started to question whether a sub-tree was the right way to go, because that would mainly be for @AndreasArvidsson to use, as everyone else is on a fork, so wouldn't use the subtree, and realistically we imagined @AndreasArvidsson wouldn't last very long without changing it. Even in the case of the above formatters, it seems unlikely we'll come up with something @AndreasArvidsson would be willing to use without having to change the code, but we can see how far we get and if we succeed with formatters it's a good sign

[jaresty]

It would be my hope that changing the code as necessary would be feasible and that we'd all benefit from those changes. Already, I can say that these formatters changes sound really nice and I didn't have them because I was unaware they existed, so this direction is directly benefiting me-thank you.

[jaresty]

I'm not sure if this visualization is helpful, but I tried to capture the problem that I'm seeing in a diagram in case the visual representation helps. I could certainly be missing some things. This seems to be what's going on to me:

I think that the feedback to the community repository is being hampered by the fact that many are using forks. That also in turn exacerbates the problem. There's a alternative reality that could be positive for the community repository that looks a little bit like this:

At least that's how I'm thinking about it right now. I'd love to discuss more, especially if you see it differently, because I'm sure I could learn from your experience.