Deleting history

[ellie]

So this has been proposed for a super long time.

It's taken a while, partly because deletion with sync is difficult. Deleting on one machine would result in it coming back the next time a sync happens, so we need to ensure deletion across all machines

I'm starting a new issue to track this, as it's actively being worked on

I wrote up my plan in a comment on #70, but copying it here for clarity

---

tl;dr for the plan, which I'd like to break down a bit more and HOPEFULLY get in for a v12 release. Or at least partly!

Deleting things, finally, as everyone has been asking me for it

(cue brain dump)

1. Switch to event log sync

I started on this here: https://github.com/ellie/atuin/pull/390

The difficulty here is that actually deleting the entire row will change the count of rows. Syncing two append-only logs is much much easier and simpler than syncing two arbitrary length sets of data, especially when each installation of atuin is both read and write (so technically multi-master).

The premise with this solution is that instead of syncing up history items, as we do at the moment, we sync up events. I'm proposing two kinds of event. A CREATE event, and a DELETE event. For backwards compatibility, we'd want to add a CREATE event for every currently-existing item of history.

To delete an item of history, we push up a DELETE event to the log. This is still only an append, so we can retain our simple sync logic + keep things nice and easy.

If we replay the log on a fresh client, the history item will end up being created + then deleted. Mostly there, and mostly achieves the use case of "I don't want this thing showing up in my history".

Once that works, we could expand it to also blank out the data section of a CREATE event when a DELETE event has been created, though that would cause limitations in future sync methods (I was conisdering something based on merkle trees).

2. Implement UI for deletion

I'd like this to be smooth, obvious, and not some arbitrary key combination. But also not easy to do by accident! We may also want the option to delete a single occurence of a command, but not delete all historical occurances of it (and vice-versa).

To allow for this, it would be good to be able to open a "context menu" on a single item of history. I'd suggest using the tab character for this.

Scroll back to the item you wish to delete. Hit tab, and we switch to the menu item for that history. As well as showing extra information about it (directory executed in, some other context, etc), we can have an option for deleting it!

Might want some kind of UI hint to make it obvious that tab does stuff

3. done...?

Probably not, I imagine some things will break. I'd like to roll this out gradually, as we need to maintain backwards compatibility. Will probably save old style history rows and event rows side-by-side, and start by just syncing CREATE events up and not even implementing DELETE until the next version.

[ellie]

Just merged the first part of this!

https://github.com/ellie/atuin/pull/390

You should see a new table created locally + tracking events

[tobiasvl]

I'd like this to be smooth, obvious, and not some arbitrary key combination. But also not easy to do by accident!

I'd personally love it if I could just highlight the entry and simply press Del to delete it, or perhaps Shift+Del like Chrome does it in the URL bar. This seems smooth and obvious and non-arbitrary to me, and especially the latter one is hard to do by accident.

[eripa]

In addition to selectively deleting records it would be good to have a way to configure sort of retention or max age/limit. I.e. "remove any history items older than 5y or 500k lines" or similar.

[ellie]

In addition to selectively deleting records it would be good to have a way to configure sort of retention or max age/limit. I.e. "remove any history items older than 5y or 500k lines" or similar.

For sure! I've been thinking about that too. It has the same problem of breaking sync though 🙃

[manfredlotz]

From a usage point of view I think the UI part for deleting is not so important.

When wanting to delete something in history I

• would like to check what to delete by running atuin search <srchpat1> ...
• and then deleting those entries by running atuin delete <srchpat1> ...

PS: For me the ability to delete items in the history is also very important.

[mentalisttraceur]

You might find it helpful to

1. look at what people have figured out in shared text editing (whether multi-user or single user racing with async editor plugins), because they have to deal with what is fundamentally the same problem; and

2. also look at how undo is implemented, because a deletion is in many ways like an undo.

I remember reading some blog posts about implementing the Xi editor a few years ago were fairly enlightening for me, which might be relevant here. In particular this one about CRDT and async undo.

[ellie]

From a usage point of view I think the UI part for deleting is not so important.

When wanting to delete something in history I

* would like to check what to delete by running `atuin search <srchpat1> ...`

* and then deleting those entries by running `atuin delete <srchpat1> ...`

PS: For me the ability to delete items in the history is also very important.

Super true. I'll probably release this first and think about the UI later

[ellie]

You might find it helpful to

1. look at what people have figured out in shared text editing (whether multi-user or single user racing with async editor plugins), because they have to deal with what is fundamentally the same problem; and

2. also look at how _undo_ is implemented, because a deletion is in many ways like an undo.

I remember reading some blog posts about implementing the Xi editor a few years ago were fairly enlightening for me, which might be relevant here. In particular this one about CRDT and async undo.

I did spend a while reading this literature + thinking about it, but they have a much harder problem to solve - that of multiple users + edit conflicts.

Neither of those problems really apply here, we just want to append stuff - and that can either be a CREATE or a DELETE

[mentalisttraceur]

@ellie Cool, sorry it ended up not relevant. Been a few years since I read those.

Was just throwing it out there in case those more general solutions (or the libraries that implement them) for distributed edits ended up being a good fit.

Because to me a synced create+delete log felt like the beginnings or special case of the same problem shape, and in your idea of then going back to erase the data in the create entry I saw some (very superficial/abstract) parallels to how the Xi editor design docs mentioned leaving tombstone entries for deleted text.

[mentalisttraceur]

I also think it's relevant because I feel like Atuin's success (and forecast of more success) is big enough that desire (or even need, maybe) for more complex history editing is just around the corner.

For example, the Xi blog post talks a bit about undo groups, and I wouldn't be surprised if eventually Atuin's community grows plugins that add functionality on top of Atuin, and then that leads to things like "would be great if I could sync a transaction of history edits, so that the first edit only goes through if another edit also goes through", and so on.

Of course we should not let that get in the way of building something that meets current needs now.

But if you like the idea then perhaps it doesn't need to get in the way - perhaps it ends up being similarly easy to implement (with so much literature to guide it), or perhaps we can end up finding a good reusable library out of the core of some multi-user/async editor.

Not trying to talk you into something you don't want to do though! If you already have vision/inertia/motivation for the simple create+delete log approach, or the CRDT/etc approach doesn't seem like the right move at least at this time, 👍

[ellie]

I saw some (very superficial/abstract) parallels to how the Xi editor design docs mentioned leaving tombstone entries for deleted text.

For sure! It actually has a lot of similarities with some other data problems too - eg the clickhouse mergetree.

I feel like Atuin's success (and forecast of more success)

Haha thank you for the positive energy 🚀

Not trying to talk you into something you don't want to do though! If you already have vision/inertia/motivation for the simple create+delete log approach, or the CRDT/etc approach doesn't seem like the right move at least at this time, 👍

So the idea here is that something like a CRDT would exist one layer of abstraction above this. With any CRDT, you need a mechanism for actually syncing the communication/data behind it. The event sync would be that - there's nothing stopping anyone in the future from implementing a CRDT on top of this. The events as currently written are basically just the same as the history blob sync, except they're just tagged with a type. The event sync doesn't care if it's a create, a delete, undo, or some other future event type - it's only when we actually process the events that this matters, and having something like a CRDT may be useful in the future!

I'm hoping with this approach we can solve our current problems, but still allow for future flexibility.

[hlascelles]

Thanks for this @ellie! I can use delete with atuin search "foo" --delete... But is there a keybinding to delete a row when looking at the main atuin UI?

I may be searching for something, and see an out of date or sensitive line, and just want to highlight it and some key to remove it from history. Has that been discussed? I can open a new issue if needed. Thanks!

[bvergnaud]

Hey, not Ellie here. 😁

In the current version of atuin, in the main UI you can hit Ctrl+o to inspect the highlighted entry, and in that new view, you can hit Ctrl+d to delete it. Hope that helps. 😉

[hlascelles]

Thank you @bvergnaud, that's the one! :bowing_man:

Neither CTRL+O keybinding or the CTRL+D are mentioned in the keybindings documentation: https://docs.atuin.sh/configuration/key-binding/#atuin-ui-shortcuts

Searching for "delete" in there doesn't reveal it either.

Should I do a PR for all that in https://github.com/atuinsh/docs ?

[bvergnaud]

Thank you @bvergnaud, that's the one! 🙇‍♂️

Neither CTRL+O keybinding or the CTRL+D are mentioned in the keybindings documentation: https://docs.atuin.sh/configuration/key-binding/#atuin-ui-shortcuts

Searching for "delete" in there doesn't reveal it either.

It's a pretty recent feature, it was added in v18 in early february. I guess the documentation was forgotten.

Should I do a PR for all that in https://github.com/atuinsh/docs ?

I don't want to speak for the maintainers here but if you've got the time for it, I'm sure they'll appreciate it. 🙂