search

chylex / Discord-History-Tracker

Desktop app & browser script that saves Discord chat history into a file, and an offline viewer that displays the file.
https://dht.chylex.com
MIT License
475 stars 83 forks source link

Desktop app keeps getting disconnected #168

Closed SageSystems closed 2 years ago

SageSystems commented 2 years ago

after around 10k messages the Desktop app script will stop recording messages and report "disconnected" and will need to be restarted using the linux build and linux desktop app/chromium/firefox

TheTechRobo commented 2 years ago

I sometimes have luck stopping and starting tracking from the desktop app.

chylex commented 2 years ago

I'll be releasing an update soon that has some general improvements, I don't know if they will help in your case but it's worth a try. You can also try turning off image previews in Discord settings.

The disconnect happens if your browser is sending tracked messages to the app, and the app doesn't respond in a few seconds and assumes the app has been stopped/crashed. Can you check the Network tab in the discord app/browser dev tools, click XHR to enable a filter, and see if there are requests like track-messages that take several seconds and eventually turn red after a timeout? Don't send any screenshots, this tab can expose your authentication token and other sensitive data, but it's a good tool for seeing what's going on.

chylex commented 2 years ago

Update released, try and see if it helps. On my laptop I can track tens of thousands of messages without ever disconnecting, but maybe a slower machine will still need longer timeouts.

TheTechRobo commented 2 years ago

I find after about 2 million messages on my hard drive it starts happening and requires a restart. On my NVMe SSD it takes more like 5 million.

chylex commented 2 years ago

Interesting, I'll try filling up a database with huge random data and see if I can improve the performance.

TheTechRobo commented 2 years ago

Something interesting to note: At least on my hard drive (I never paid attention to the logs on my NVMe drive, since it was so much rarer) the logs show that the request was received and being acted on, but something (probably the actual I/O to the database) causes the response to stall and the userscript reports "disconnected".

chylex commented 2 years ago

Most likely. The tracking script does not continue autoscrolling until the database is written to, so that it doesn't accidentally skip past messages if the requests start suddenly failing.

chylex commented 2 years ago

Once I get to several million messages it's taking a while to refresh the amount of messages in the status bar and Viewer tab, taking >1s on an NVMe drive. Can totally imagine this causing problems on hard drives with way fewer messages. I have committed solutions to various issues I found, that altogether should eliminate this problem.

chylex commented 2 years ago

Update released.

drakou111 commented 2 months ago

I am still getting this error. It happens if there are a lot of links / videos / images within the 50 messages it's trying to get, which takes longer than expected and disconnects. The 'track-users' stops after 5.00s. I think a way to fix this would either be to allow a variable amount of timeout in the advanced settings, or make it auto-retry if it disconnects.

chylex commented 1 month ago

I am still getting this error. It happens if there are a lot of links / videos / images within the 50 messages it's trying to get, which takes longer than expected and disconnects. The 'track-users' stops after 5.00s. I think a way to fix this would either be to allow a variable amount of timeout in the advanced settings, or make it auto-retry if it disconnects.

It's because downloads are being streamed directly into the database, and any delays in the streaming seem to block other database transactions from running.