Init takes forever

[jmauro]

Hello,

First, I just want to thank you for this tremendous tools!!! ❤️

I am encountering a small issue when trying to initialize the tool with my secret key:

hishtory init "XXXXXXXXXXXXXXXXXXXXXXXX"
Your current hishtory profile has saved history entries, are you sure you want to run `init` and reset?
Note: This won't clear any imported history entries from your existing shell
[y/N]y
Setting secret hishtory key to XXXXXXXXXXXXXXXXXXXX
time="2024-02-02T11:52:39Z" level=info msg="ApiGet(\"/api/v1/register?user_id=XXX=&device_id=XXX\"): 615.651926ms\n"
Killed
$ hishtory status -v
hiSHtory: v0.266
Enabled: true
Secret Key: XXXXXXXXXXXXXXXXXXXXXXXXXX
User ID: XXXXXXXXXXXXXXXXXX
Device ID: XXXXXXXXXXXXXXXXXXX
Sync Mode: Enabled
Sync Status: Synced
Commit Hash: 81ed634a4baa0414639f102a308e21907f68f703

And to be honest, I am unable to provide any logs apart from what is present in this comment. If you need my secrete key to troubleshoot, just drop a PM.

Kind regards, JM

[jmauro]

I found the issue, the process was oom-killed. Sorry for the noise.

BTW, the init could take up to 3Go in memory 😉

[jmauro]

Since I was able to debug a bit more, I found out that the init process consumes a lot of memory (At peak time, my init was consuming 4,5 Go):

I am not sure if this could be avoided (I might have a lot of history to import).

Kind regards, JM

[ddworken]

Thank you for opening this issue! You must have quite the large history file--can you share how many lines are in it (i.e. history | wc -l)? I've started to work on optimizing this in #171, but I want to continue to iterate and benchmark this to ensure that it is a solid long term solution.

[jmauro]

Thank you for opening this issue! You must have quite the large history file--can you share how many lines are in it (i.e. history | wc -l)? I've started to work on optimizing this in #171, but I want to continue to iterate and benchmark this to ensure that it is a solid long term solution. Hello,

I am around 5000 entries... Not that big:

ls -al ~/.hishtory/
total 61508
drwxr--r--  2 root root     4096 Feb  6 17:05 .
drwx------ 11 root root     4096 Feb  6 16:03 ..
-rw-r--r--  1 root root      800 Feb  6 17:05 .hishtory.config
-rw-r--r--  1 root root  8183808 Feb  6 17:03 .hishtory.db
-rw-r--r--  1 root root    32768 Feb  6 17:05 .hishtory.db-shm
-rw-r--r--  1 root root  4177712 Feb  6 17:05 .hishtory.db-wal

[GRbit]

I don't know if it helps, but I noticed similar behavior, but it differs from case to case. I added around 5 machines, having 30-50k history file on two of them. The largest hist files machines went first, than I added the smaller ones. On the smaller ones, sometimes init was quite fast, but twice it took quite a time.

Hope it'll give some valuable context to this bug.