Open dflock opened 3 years ago
Thank you for your report! Did you see any message like McFly: Importing shell history for the first time. This may take a minute or two...
in your shell?
I just pushed up a change, would you mind checking if that helped?
No, didn't get any messages, just sat there grinding the disk and not returning to the prompt.
OK, giving it a try. I got the McFly: Importing shell history for the first time. This may take a minute or two...
message this time
I'll let it run and tell you how long it takes :)
I hope it's not more than a couple minutes.
Well... it took from 22:29:55 until 23:22:12, almost an hour.
$ ll ~/.mcfly/history.db
Permissions Size User Group Date Modified Name
.rw-r--r-- 1.7M duncan duncan 12 Mar 23:22 /home/duncan/.mcfly/history.db
Not played with it much yet, but it seems to work OK now.
I'm glad it finished, but am surprised it took so long! What kind of computer do you have?
OS: Xubuntu 20.04.2 LTS x86_64
Kernel: 5.4.0-66-generic
Uptime: 17 days, 17 hours, 3 mins
Packages: 68690 (apt), 53 (flatpak), 1 (snap)
Shell: bash 5.0.17
Resolution: 1920x1080, 1920x1080, 1920x1080
DE: Xfce 4.16
WM: Xfwm4
WM Theme: Arc-Dark
Theme: Arc-Dark [GTK2/3]
Icons: Papirus-Dark [GTK2/3]
Terminal: kitty
Terminal Font: Operator Mono SSm Lig Book
CPU: Intel i7-6700K (8) @ 4.200GHz
GPU: Intel HD Graphics 530
Memory: 38005MiB / 64206MiB
I know it's a fairly large history file, but this workstation should be able to process 26k lines of text in a few seconds, at the very most.
Agreed, I think this import process needs to be profiled and optimized.
Also, MCFLY_HISTORY_LIMIT
doesn't currently affect import, so it makes sense that it didn't help. Should it?
Also, MCFLY_HISTORY_LIMIT doesn't currently affect import, so it makes sense that it didn't help. Should it?
It shouldn't be necessary at all, really.
Agreed. This should be optimized.
I had the same issue and adding a transaction around the insert (#146) fixed it for me. Maybe this can be considered fixed? @dflock can you test with the latest master
to see if it's improved?
I installed today and the initial import took 12s on a 13 Mb history file.
🕙 2021-08-02 11:10:01 ❯ source ~/.bashrc
McFly: Importing shell history for the first time. This may take a minute or two...done.
✦2 🕙 2021-08-02 11:10:13 ❯ ls -al ~/.mcfly/history.db ~/.dotfiles/bash/.bash_eternal_history
.rw-r--r-- michaeljoseph staff 13 MB Mon Aug 2 11:15:34 2021 /Users/michaeljoseph/.dotfiles/bash/.bash_eternal_history
.rw-r--r-- michaeljoseph staff 44 MB Mon Aug 2 11:15:34 2021 /Users/michaeljoseph/.mcfly/history.db
I noticed that running any command in my shell became painfully slow. After some testing, realized that moving the history.db file fixed the issue.
the db was 13M
eval "$(mcfly init zsh)"
export MCFLY_KEY_SCHEME=vim
export MCFLY_FUZZY=2
export MCFLY_RESULTS=2048
export MCFLY_HISTORY_LIMIT=10000
export MCFLY_INTERFACE_VIEW=BOTTOM
export MCFLY_RESULTS_SORT=LAST_RUN
My DB is 18M and I haven't had any issues. What kind of computer do you have? It might be nice for us to add execution time logging to the shell scripts.
@cantino Hello, first of all, thank you for your work to bring us this great tool! I am facing the same problem and I have found reproducible steps (only for my environment).
When I was testing the curl command, I accidentally copied the browser's interface for uploading images as a curl command and pasted it into my terminal, which caused .zsh_history
to record this command with binary text, and from then on, my terminal always became abnormally slow, and the only way for my terminal to return to normal was to delete this line with binary text from .zsh_history
or remove mcfly from .zshrc
to get my terminal back to normal.
I tried copying the command directly into the github input box, but it seems to cause the page to crash (WTF), so I'm writing this one .zsh_history
record to a txt file and uploading it, in the hope that this method will replicate the problem and not just be my personal problem!
Thank you @hexh250786313, appreciated! I suspect this is due to an incredibly long line (4523740 bytes) in the history and how sqlite is handling it. We should add a length limit to new commands and truncate them, that would likely fix it.
I have this in my .bashrc:
Doing
. ~/.bashrc
takes ~forever. I have no idea how long, because I did ctrl+c after a minute or so. This is a new install. I've tried changingMCFLY_HISTORY_LIMIT=1000
, doingrm ~/.mcfly/history.db
, etc... still takes ages/doesn't complete. Lots of disk i/o too.I assume this is because I have a large history file. These are my current bash history related settings:
My
~/.bash_eternal_history
file is currently 497.5 KiB, 26,339 lines. It also includes timestamps, so the last bit of it looks like this:If I do this, mcfly starts working: