Network disconnect resiliency

jojo2357 / kiwix-zim-updater

A script to check `download.kiwix.org` for updates to your local ZIM library.

GNU General Public License v2.0

77 stars 5 forks source link

Network disconnect resiliency #16

Closed jojo2357 closed 1 year ago

jojo2357 commented 1 year ago

Changelog:

Use some lower-overhead api calls (that is why ?F=0 exists)
Add option to skip updating the script.
Add option to verify library integrity. This only works for newer files, and we could make it work for older files if we cache the sha's (which are absolutely tiny) but that wont help if they didnt use this script to dl it, or the shas are missing.
Remove curl dependency. Downloads zims with wget. Wget is more resilient out of the box.
ALLOW THE USER TO ctrl+c AND CONTINUE ANY IN PROGRESS DOWNLOAD. Use the verify checksum option and let me know if you can get a failed download

It is becoming more and more apparent that the purge and download steps need to merge. I think I will do that separately and I already have a file in progress with the changes locally

closes #15 closes #14 (maybe, if we get consensus)

DocDrydenn commented 1 year ago

Haven't reviewed yet, but I did notice one of the changes: replacing curl with wget.

When I first wrote this script, I used wget. I switched over to curl because wget wasn't able update the log files in "real-time". This script was originally intended to be run as a cronjob in the background... with the ability to be run manually as well.

If run in the background, there would be no way to "monitor" the current status of any downloads. 'curl' was able to update the log files in real-time, so I switched. This way, one only needed to tail the log file to see the download progress/status/etc... in real-time. Do you know of a way to make wget do the same?

I am not opposed to loosing this "real-time" monitoring feature for the betterment of the script, however, it was a requested feature by a user via Reddit back when I first released/announced it.

jojo2357 commented 1 year ago

I will look into it. the progress bar does appear in the terminal now, albeit with some different info.

I remember seeing this SO which suggested instead to manually create the progress bar, which would in theory make that very easy.

More research is needed

jojo2357 commented 1 year ago

@DocDrydenn Done in 59bf86b7d9f82848a4b6440d5da08fbc569d3c6e

DocDrydenn commented 1 year ago

Sorry, I won't be able to check any of this until later tonight...

I remember playing around with the progress bar in wget... problem was that it didn't update the log file in real-time... it only updated after the download completed... which was no good for external monitoring... Is this no longer the case?

jojo2357 commented 1 year ago

Affirmative. Progress bar in the log updates via tee -a just as curl did

DocDrydenn commented 1 year ago

Thoughts on creating a verification log file? ... like the download and purge log files? ... something that could be checked after a library verification if the script is run in the background/via cron?

jojo2357 commented 1 year ago

Might be good to understand if a lot of updates fail...i still am yet to see the checksum not come through correctly. Anyway, should be easy to add

jojo2357 commented 1 year ago

Did you test it before merging?

DocDrydenn commented 1 year ago

I tested as best as I could and everything seemed to be working.