Performance issue?

[tmpm697]

I try to figure out what is consume time at startup of my z-shell configs, histdb took 1st place.

for i in $(seq 1 10); do time zsh -i -c exit; done

num  calls                time                       self            name
-----------------------------------------------------------------------------------
 1)    1          13.32    13.32   34.12%     13.32    13.32   34.12%  _histdb_start_sqlite_pipe

How to speed up histdb?

[larkery]

This is not a problem for me. If you can make it work better without breaking it, please submit a pull request.

[tmpm697]

Thanks, I’m using histdb command with 6k lines pulled from sqlite3 and feel pretty slow ~ half second through fzf.

Used to use 60k history file like instantly.

[larkery]

I think mine has about 100k rows at the moment, so I think something is different between our systems

$ time zsh -i -c exit                         
zsh -i -c exit  0.04s user 0.01s system 99% cpu 0.054 total

[tmpm697]

for i in $(seq 1 10); do time (histdb | tail -n +2) >/dev/null; done
( histdb | tail -n +2; ) > /dev/null  0.11s user 0.02s system 118% cpu 0.107 total
( histdb | tail -n +2; ) > /dev/null  0.11s user 0.02s system 120% cpu 0.111 total
( histdb | tail -n +2; ) > /dev/null  0.09s user 0.04s system 120% cpu 0.110 total
( histdb | tail -n +2; ) > /dev/null  0.11s user 0.02s system 120% cpu 0.110 total
( histdb | tail -n +2; ) > /dev/null  0.12s user 0.01s system 119% cpu 0.110 total
( histdb | tail -n +2; ) > /dev/null  0.11s user 0.02s system 121% cpu 0.107 total
( histdb | tail -n +2; ) > /dev/null  0.12s user 0.01s system 120% cpu 0.110 total
( histdb | tail -n +2; ) > /dev/null  0.11s user 0.02s system 120% cpu 0.113 total
( histdb | tail -n +2; ) > /dev/null  0.12s user 0.01s system 119% cpu 0.108 total

This is >110ms response and >120% cpu, this is a lot. We need somehow improve query in db.

[larkery]

The reason that query is slow is because you are querying and formatting the entire database content to cut off the last 2 lines. Because the output is tabulated this involves scanning it and formatting it at least twice. The histdb command has only had attention by me for interactive use on the shell; it works fast enough for that so I don't consider this a problem.

If you want the bottom 2 rows, try histdb --limit 2. If you want to do some large batch process on the database contents, write your own query against it (have a look at the source to see how).

Regarding the idea that "We need [to] somehow improve query in db": I'm happy to review pull requests. However I don't really do feature requests or stuff like that.

The reason zsh-histdb exists at all is to store and query my shell history in a way I find useful on my machines. If it does this for other people that's nice, but not so nice where I'm motivated to spend lots of time working on it. As far as I'm concerned, it's basically feature-complete and works.

So, if you would like this to work better for you, do a bit of profiling and see if you can improve it.

[tmpm697]

Thanks, I can only think the way how to improve this issue by have histdb command work as histdb | tail -n +2.

This is for discussion if we can improve, it contains facts that can produce instead of feature request.

[larkery]

I don't understand what you have written I'm afraid