cantino / mcfly

Fly through your shell history. Great Scott!
MIT License
6.85k stars 178 forks source link

Really slow startup with very long line in history #131

Open dflock opened 3 years ago

dflock commented 3 years ago

I have this in my .bashrc:

export MCFLY_HISTORY_LIMIT=10000
eval "$(mcfly init bash)"

Doing . ~/.bashrc takes ~forever. I have no idea how long, because I did ctrl+c after a minute or so. This is a new install. I've tried changing MCFLY_HISTORY_LIMIT=1000, doing rm ~/.mcfly/history.db, etc... still takes ages/doesn't complete. Lots of disk i/o too.

I assume this is because I have a large history file. These are my current bash history related settings:

#
# History settings
#
shopt -s histappend # Append to the history file, don't overwrite it
shopt -s cmdhist # Save multi-line commands as one command
export HISTSIZE=-1 # Keeps all history
export HISTFILESIZE=-1 # Infinite file size
export HISTTIMEFORMAT='%F %T ' # Useful timestamp format
export HISTIGNORE="&:[ ]*:exit:clear"
export HISTFILE=~/.bash_eternal_history
# Record each line as it gets issued
[[ "$PROMPT_COMMAND" == *'history -a'* ]] || export PROMPT_COMMAND="history -a;$PROMPT_COMMAND"

My ~/.bash_eternal_history file is currently 497.5 KiB, 26,339 lines. It also includes timestamps, so the last bit of it looks like this:

#1615575032
. ~/.bashrc
#1615575046
rm ~/.mcfly/history.db
#1615575049
micro ~/bash/mcfly.bash
#1615575057
micro ~/.bashrc
#1615575459
micro ~/bash/env.bash

If I do this, mcfly starts working:

$ mv ~/.bash_eternal_history ~/_.bash_eternal_history
$ rm ~/.mcfly/history.db
$ . ~/.bashrc
cantino commented 3 years ago

Thank you for your report! Did you see any message like McFly: Importing shell history for the first time. This may take a minute or two... in your shell?

cantino commented 3 years ago

I just pushed up a change, would you mind checking if that helped?

dflock commented 3 years ago

No, didn't get any messages, just sat there grinding the disk and not returning to the prompt.

dflock commented 3 years ago

OK, giving it a try. I got the McFly: Importing shell history for the first time. This may take a minute or two... message this time

I'll let it run and tell you how long it takes :)

cantino commented 3 years ago

I hope it's not more than a couple minutes.

dflock commented 3 years ago

Well... it took from 22:29:55 until 23:22:12, almost an hour.

$ ll ~/.mcfly/history.db

Permissions Size User   Group  Date Modified Name
.rw-r--r--  1.7M duncan duncan 12 Mar 23:22  /home/duncan/.mcfly/history.db 

Not played with it much yet, but it seems to work OK now.

cantino commented 3 years ago

I'm glad it finished, but am surprised it took so long! What kind of computer do you have?

dflock commented 3 years ago
OS: Xubuntu 20.04.2 LTS x86_64 
Kernel: 5.4.0-66-generic 
Uptime: 17 days, 17 hours, 3 mins 
Packages: 68690 (apt), 53 (flatpak), 1 (snap) 
Shell: bash 5.0.17 
Resolution: 1920x1080, 1920x1080, 1920x1080 
DE: Xfce 4.16 
WM: Xfwm4 
WM Theme: Arc-Dark 
Theme: Arc-Dark [GTK2/3] 
Icons: Papirus-Dark [GTK2/3] 
Terminal: kitty 
Terminal Font: Operator Mono SSm Lig Book 
CPU: Intel i7-6700K (8) @ 4.200GHz 
GPU: Intel HD Graphics 530 
Memory: 38005MiB / 64206MiB 
dflock commented 3 years ago

I know it's a fairly large history file, but this workstation should be able to process 26k lines of text in a few seconds, at the very most.

cantino commented 3 years ago

Agreed, I think this import process needs to be profiled and optimized.

cantino commented 3 years ago

Also, MCFLY_HISTORY_LIMIT doesn't currently affect import, so it makes sense that it didn't help. Should it?

dflock commented 3 years ago

Also, MCFLY_HISTORY_LIMIT doesn't currently affect import, so it makes sense that it didn't help. Should it?

It shouldn't be necessary at all, really.

cantino commented 3 years ago

Agreed. This should be optimized.

SafariMonkey commented 3 years ago

I had the same issue and adding a transaction around the insert (#146) fixed it for me. Maybe this can be considered fixed? @dflock can you test with the latest master to see if it's improved?

michaeljoseph commented 3 years ago

I installed today and the initial import took 12s on a 13 Mb history file.

🕙 2021-08-02 11:10:01  ❯ source ~/.bashrc 
McFly: Importing shell history for the first time. This may take a minute or two...done.

✦2 🕙 2021-08-02 11:10:13  ❯  ls -al ~/.mcfly/history.db ~/.dotfiles/bash/.bash_eternal_history
.rw-r--r-- michaeljoseph staff 13 MB Mon Aug  2 11:15:34 2021  /Users/michaeljoseph/.dotfiles/bash/.bash_eternal_history
.rw-r--r-- michaeljoseph staff 44 MB Mon Aug  2 11:15:34 2021  /Users/michaeljoseph/.mcfly/history.db
Bad3r commented 1 year ago

I noticed that running any command in my shell became painfully slow. After some testing, realized that moving the history.db file fixed the issue. 

the db was 13M 

eval "$(mcfly init zsh)"

export MCFLY_KEY_SCHEME=vim

export MCFLY_FUZZY=2

export MCFLY_RESULTS=2048

export MCFLY_HISTORY_LIMIT=10000

export MCFLY_INTERFACE_VIEW=BOTTOM

export MCFLY_RESULTS_SORT=LAST_RUN  
cantino commented 1 year ago

My DB is 18M and I haven't had any issues. What kind of computer do you have? It might be nice for us to add execution time logging to the shell scripts.

hexh250786313 commented 2 weeks ago

@cantino Hello, first of all, thank you for your work to bring us this great tool! I am facing the same problem and I have found reproducible steps (only for my environment).

When I was testing the curl command, I accidentally copied the browser's interface for uploading images as a curl command and pasted it into my terminal, which caused .zsh_history to record this command with binary text, and from then on, my terminal always became abnormally slow, and the only way for my terminal to return to normal was to delete this line with binary text from .zsh_history or remove mcfly from .zshrc to get my terminal back to normal.

I tried copying the command directly into the github input box, but it seems to cause the page to crash (WTF), so I'm writing this one .zsh_history record to a txt file and uploading it, in the hope that this method will replicate the problem and not just be my personal problem!

mcfly.txt

cantino commented 2 weeks ago

Thank you @hexh250786313, appreciated! I suspect this is due to an incredibly long line (4523740 bytes) in the history and how sqlite is handling it. We should add a length limit to new commands and truncate them, that would likely fix it.