Add `history search`, `history delete`, `history tag`

[CJ-Wright]

Would it be possible to

1. set history up so that we can slice through the various saved histories (eg history -1 gives the latest history file)
2. search history for keys in the history. This could be super helpful when you don't really remember what you did well enough to go get it from the history (eg history search raumel.yaml in out)

For community

⬇️ Please click the 👍 reaction instead of leaving a +1 or 👍 comment

[ghost]

atm history -1 returns the last command

[CJ-Wright]

Sorry maybe they're should be a second clause to that command (the proposed syntax were mostly there to help try to explain the request, less as a specific syntax proposal)

[AstraLuma]

There should probably be an actual history API with a coherent, abstracted model for history.

[scopatz]

@CJ-Wright @astronouth7303 does this help - http://xon.sh/api/history/index.html?

[scopatz]

Also, you can access the current history object via __xonsh__.history

[AstraLuma]

Sorta? I suspect that API exposes too much of the internals and handling multiple backends intelligently is not abstracted away.

Basically, an interface that we can say "Yes, Users, this is how you manipulate your history" and not have to apologize for.

[scopatz]

The base class methods should be pretty general, but you are right that the search stuff is not in the base presently. It could use improvements! Happy to review PRs!

[CJ-Wright]

I think that the search support is key, that way we can find the history we are looking for, rather than hoping to find the right uid.

[AstraLuma]

Sounds like something we need to have an actual discussion about. What kind of searching do we support? Full text? Substring? Prefix? Regex? Is whatever searching we decide efficiently implementable with conceivable backends?

[CJ-Wright]

We might also discuss how to put additional metadata inside the initial data (not the cmds) so that it is easier to search. Eg I run some data analysis inside of xonsh and it turns out really good, I'd like to tag my session with something so I can find it again.

[AstraLuma]

@CJ-Wright See how many different use cases you can come up with for non-command entries in history.

[CJ-Wright]

Tags:

1. "Crystallography Problem Set 5"
2. "Update system packages"
3. "Data processing on x-ray data from beamtime"
4. "Better data processing on x-ray data from beamtime"
5. "Test out new configuration"
6. "Come back to this when PR gets merged"

The list is really as long as your imagination lets it be. It might be nice to be able to build a narrative of what you did with history, and part of that narrative is your own view of why you did things or other things you think you should keep track of beyond just what was run and when. This could be crucial if you wanted to build a mechanism for reporting what you did in a terminal (eg I use this as my data introspection tool and would like a nice LaTex report of what I did, with some narrative in there too. Or I did some work on the servers I manage but something feels odd I should try to remember this for latter).

I guess one could argue that if you just wrote this into the command line it would show up in the search, but then you could say that about all the top level metadata that is stored in the history documents.

[AstraLuma]

How about things besides tags? How structured should tags be?

Could some of these use cases be covered by temporarily inserting a logger backend? eg, if you want to make a report of your session, you could say record begin and then record finish <filename> (or whatever) and it would use the history hooks/process events to get the commands and their results. (In a hypothetical world were we had better management of multiple backends.)

[CJ-Wright]

I can't think of too many things off the top of my head other than tags (although others might). I'm mostly inspired by http://nsls-ii.github.io/databroker/headers.html Figuring out the structure could be a bit tricky. Currently I'd just say to roll them into the top level metadata (next to env, ts, etc.) as key: value pairs with the expectation that the user will provide the keys and values (although we could hint at some common names, like name, purpose, reason, etc.) This is how it is currently done at the NSLS-II (see above link). There is nothing stopping the user from putting in more complex data structures into the keys or values (with the exception that the keys must be hashable?) although searching through those could be more tricky (see mongo's language for searching inside of objects).

You could have a logger backend. Although it might be easier at that point to just inspect the list of commands and slice to then export. The main point was this kind of user defined data is useful for building a report.

If anyone likes the databroker model I can try to explain it more (including the internals). It seems to do a good job at handling heterogeneous data (like this data).

Edit: didn't complete my thought

[AstraLuma]

I mean, just a string marker? Start/stop? Heirarchal tags? Nested? Unnested?

My suggestion to use a backend is that if you know you're going to do the equivalent to script, why not get the data directly? (Especially since the history backend may strip output, or maybe other mangling?)

[paidhi]

Something like history search and history delete the way Fish provides this would be nice.

history search [ --show-time ] [ --case-sensitive ] [ --exact | --prefix | --contains ] [ --max=n ] [ --null ] [ "search string"... ]
history delete [ --show-time ] [ --case-sensitive ] [ --exact | --prefix | --contains ] "search string"...

[anki-code]

Something like history search and history delete

If you're using SQLite history backend the way to query history is very simple:

# search
sqlite3 $XONSH_HISTORY_FILE "SELECT inp FROM xonsh_history WHERE inp LIKE 'cd%' ORDER BY tsb DESC LIMIT 10"
# as well as macro call:
sqlite3 $XONSH_HISTORY_FILE ! SELECT inp FROM xonsh_history WHERE inp LIKE 'cd%' ORDER BY tsb DESC LIMIT 10

# delete
sqlite3 $XONSH_HISTORY_FILE "DELETE FROM xonsh_history WHERE inp LIKE '%secret_key_in_command%'"

You can also use REGEXP, GLOB, MATCH instead of LIKE.

So you can create simple ExecAlias:

aliases['history-search'] = """sqlite3 $XONSH_HISTORY_FILE @("SELECT inp FROM xonsh_history WHERE inp like '%"+$arg0+"%' ORDER BY tsb DESC LIMIT 10") """

history-search cd
# cd /
# cd /tmp
# ...

# And also use it as macro call:
history-search! cd /home
# cd /home/user/tmp
# cd /home/user/.local
# ...

Also the history-pull allows to add the history from another sessions into the current session. Search is working as well.