search

kiwix / mirrors-qa

Q/A tools for Kiwix Download Mirrors
GNU General Public License v3.0
1 stars 0 forks source link

Update mirrors DB #12

Closed rgaudin closed 5 months ago

rgaudin commented 5 months ago

We need to initialize our DB (#6) with mirrors information and we'll also need to update them periodically. For this, we should rely on the online list of mirrors that we have.

https://download.kiwix.org/mirrors.html

It's in HTML format but all information can be extracted easily.

Let's use the URL's hostname as ID for now and also assume that any DB mirror not found in the page should be marked disabled

elfkuzco commented 5 months ago

What do you think about providing a command line interface for this?

backend update-mirrors

This way, it can be run externally by an admin or scheduled to be run periodically via cron

rgaudin commented 5 months ago

Sure! As long as we also have the ability to run it via code (still not clear how this will be trigered in prod)

elfkuzco commented 5 months ago

Perhaps, in the code, it can check if there are any rows in the table. If there are none, it will involve the function behind the CLI. Subsequent updates would be done externally or via a scheduler

rgaudin commented 5 months ago

👍

rgaudin commented 5 months ago

We will need to exclude the israeli mirror (it doesn't have the speedtest file)