Provide a way to remove commands from history

[horeah]

Sometimes it is desirable to delete some earlier command from the history (e.g. it might contain sensitive information, or maybe it's just plain wrong and we don't want it to "pollute" the history).

This action could be triggered by some key combination (Ctrl-Shift-K?) while navigating the history (either via the current paradigm "type filter string, then Up" or possibly during incremental search as proposed by #1).

Implementing this is complicated because it requires cooperation between multiple running PyCmd instances: deleting from the current instance is not enough, as the other instances will still have it and save it in the history file. This is why the only manual workaround is currently to close all PyCmd processes for the current user, then manipulate the %APPDATA%\history file using some editor.

Ideas on how to implement this (ideally: in a simple way) are welcome :smile:

[ufo]

Here comes an additional approach resp. idea for removing command history entries, which originates from another idea I had for PyCmd some day:

1. I additionally have a "history.bat" file in my PATH for some years now, which simply does this:

   @type %APPDATA%\PyCmd\history

   That way you can search N commands in command history like you're used to do on Linux, e.g.:

   history | findstr "something"

   Which gives you a nice search result overview sometimes and makes searching easier. I also love the UP+DOWN key search feature of PyCmd and intensively use it. But they are both slightly different use cases.

2. So maybe just implement a 'history' command directly in PyCmd (which can optionally be activated in init.py). So there's no need for a history.bat file.

Now if this optional internal 'history' command would behave like on Linux it would: a) Prepend line numbers before each command b) Know options like "-d" to remove specific entries from history, e.g. "history -d 123"

PS: Thus maybe also implement the Linux shell feature "!" (plus history line number) and activate it if 'history' is activated in init.py.

history
1 cd abc
3 cd ..
!1

[horeah]

1. I additionally have a "history.bat" file in my PATH for some years now, which simply does this: 

@type %APPDATA%\PyCmd\history

That way you can search N commands in command history like you're used to do on Linux, e.g.:

history | findstr "something"

This is a cool idea!

2. So maybe just implement a 'history' command directly in PyCmd (which can optionally be activated in init.py). So there's no need for a history.bat file.

As long as the functionality can be easily implemented as an external command (e.g. batch file), I am not inclined to move it inside PyCmd.

Now if this optional internal 'history' command would behave like on Linux it would: a) Prepend line numbers before each command This can also be achieved with an external script (e.g. "cat -n")

b) Know options like "-d" to remove specific entries from history, e.g. "history -d 123"

This is a reasonable interface, but I am hoping we can work out (possibly: in addition) something more user-friendly/interactive interface.

PS: Thus maybe also implement the Linux shell feature "!" (plus history line number) and activate it if 'history' is activated in init.py.

history
1 cd abc
3 cd ..
!1

I am not sure that this is feasible in the general case; PyCmd actually uses cmd.exe to execute all but the simplest commands; and parsing the cmd syntax to ensure that we correctly replace the ! in complex command is virtually impossible (this is not an exaggeration; the syntax is extremely context-heavy and has a huge number of exceptions and special cases). Pragmatically though, we might be able to live with the approximate solution we already use for ~.

But: while we can argue about the interface and come up with pros and cons for several approaches (e.g. "history command" vs "manipulation during interactive search"), I still have a hard time coming up with a simple but robust implementation of actually deleting a command from all the running PyCmd instances -- deleting from just the current instance is not enough, other PyCmd instances will just write the command back to the history file!

[horeah]

Some implementation ideas:

1. use an additional "daemon" executable to synchronize all running instances of PyCmd (this is what e.g. the fish shell is doing).

   1. PyCmd would check for a running daemon during startup, and launch it if not already running (trickier than it sounds: what if PyCmd.exe is not built into a binary but started as python PyCmd.py?)
   2. connect to the running daemon via some sort of socket/pipe (also tricky on Windows, where AF_UNIX is not supported in Python)
   3. the daemon would terminate on its own when no more PyCmd instances are connected to it
   4. PyCmd instances would notify the daemon when some "universal" operation such as history manipulation is performed
   5. Running instances listen for notifications from the daemon on a thread, and execute them as they arrive

2. write "requests" for command deletions to a dedicated file (say, %APPDATA%\PyCmd\history_delete):

   1. open the file (exclusively using e.g. a file lock)
   2. write <PID> <command-to-delete> lines inside, for each <PID> of running PyCmd instances except the current one (this is trickier than it sounds, PyCmd.exe is relatively easy but what do we do for PyCmd instances that are started via python PyCmd.py?)
   3. close the file
   4. each running PyCmd has a thread that checks this file periodically, and if it finds its PID "executes" the removal and deletes the line from the request file; it might also check whether any PID listed in the file is no longer alive, and delete that line as well.

[ufo]

This is a reasonable interface, but I am hoping we can work out (possibly: in addition) something more user-friendly/interactive interface.

I actually agree with you. When typing the idea I already thought to myself that implementing an internal 'history' command would be too much off the road somehow.

Ideas on how to implement this (ideally: in a simple way) are welcome

As silly as possible.. you asked for it 😊: Currently the history only gets updated on disc when pressing ENTER (and two more cases with ESC and CTRL+G?). Would it be too expensive resp. slow to also always update the history in memory (thus back from disc) when navigating/searching it? The current 'update_history(_to_file)' function doesn't lock the file, so I guess a new 'update_history_from_file' function wouldn't require a file lock as well, since you usually can't type into two command lines (PyCmd instances) at the same time. If this wouldn't kill the performance it would also add the advantage that all running instances would receive new commands entered into another instance. I mean without restarting the other instances. Of course it doesn't happen every day, but sometimes I miss newly entered commands in other running instances.. and then immediately remember that I would have to restart the instance in order to update the history.

[ufo]

Forget the idea about inheriting new history entries from other running instances. Neither fish nor bash behave like that. It would be too tricky and lead to unwanted behavior anyway.

[horeah]

You are making some very good points here -- thank you for looking into this topic!

I looked over PyCmd's "history update" mechanism (which was implemented many years ago) and I think we can indeed come up with a simpler implementation for deleting commands, along the lines that you sketched: when the deletion is triggered (say, Ctrl-Alt-K during history search) we remove the command from the state.history.list of the current PyCmd AND delete it from the history file as well. This would of course not delete it from the state.history.list of the other running PyCmd instances, but I think we can live with this. Some additional things to consider:

1. Pressing Esc/Ctrl-G should not actually update the history file (only the current instance's history); the idea with this feature is to be used as a handy "scratchpad" for typed commands:. My most common use case is when I start typing some command, then realize I need to run some other command first (e.g. I am in the wrong directory, or on the wrong git branch) so I can just press Esc to save the typed command into the history, run some other command, then use the Up arrow to find the original command and edit/run it.
2. Not worrying about synchronized access to the history file is only ~acceptable because PyCmd is an interactive program and we don't expect simultaneous manipulations of the history from multiple running instances. But in rare/extreme cases, delays might happen that would lead to parallel access -- it would be really good to have some synchronization here (this should probably be tracked by a separate issue)
3. We need to figure out what the state is after deleting the currently "selected" command from the history: abort the search and fallback to an empty prompt? Show the previous command? Or the next one?....

If you want to experiment with implementing this feature (it's not trivial, but hopefully also not too hard) let me know and I will assign this issue to you; if not, I will add this to my TODO list (somewhere near the top actually: this is a feature that I've been wanting to add for a long time but never realized there might be a simple implementation)

As for the live synchronization of the history across running instances: I tried to add this in at some point in the past, but it ended up more confusing than useful; also the other shells don't have this (as you mentioned) so I think we should forget about this (at least for now).

[horeah]

Starting with release 20230829, Ctrl-Alt-K (during editing, filtered search and interactive search) deletes the "current" command from the current session AND the saved history file.

Thanks @ufo for the push and the brainstorming :)