kiwix / mirrors-qa

Q/A tools for Kiwix Download Mirrors
GNU General Public License v3.0
1 stars 0 forks source link

Document mirrorbrain behavior #2

Closed rgaudin closed 5 months ago

rgaudin commented 5 months ago

In order to limit the scope of this project, and to understand what are the available variables we can act upon, we need to have a very clear understanding of how the load balancer works: what information is used and how in order to compute a request response.

I suppose it's all in mirrorbrain's documentation.

I am sharing here the configuration (removed the admin's details) as some would matter.

id identifier baseurl baseurl_ftp baseurl_rsync enabled status_baseurl region country asn prefix ipv6_only score scan_fpm last_scan comment operator_name operator_url public_notes lat lng country_only region_only as_only prefix_only other_countries file_maxsize
5 mirror.kiwix https://mirror.download.kiwix.org/ ftp://mirror.download.kiwix.org/ rsync://rsyncd-service/self.download.kiwix.org/ TRUE TRUE eu fr 0 FALSE 500 473 2023-01-27 15:44:17.793 +0000 Added - Wed Mar 8 15:11:24 2017 Kiwix https://www.kiwix.org 48.860 2.350 FALSE FALSE FALSE FALSE 0
3 mirror2 http://mirror2.kiwix.org ftp://mirror2.kiwix.org FALSE TRUE 0 FALSE 3 0 Added - Mon Apr 30 15:05:32 2012 46.000 2.000 FALSE FALSE FALSE FALSE 0
6 kiwix http://mirror.kiwix.org ftp://mirror.kiwix.org FALSE TRUE 0 FALSE 1 512 Added - Thu Jul 28 12:04:35 2011 46.000 2.000 FALSE FALSE FALSE FALSE 0
7 mirror3 http://mirror3.kiwix.org ftp://mirror3.kiwix.org FALSE FALSE eu fr 0 FALSE 10 845 Added - Thu Jan 9 20:16:22 2014 0.000 0.000 FALSE FALSE FALSE FALSE 0
8 netcologne.de http://mirror.netcologne.de/kiwix/ ftp://mirror.netcologne.de/kiwix/ FALSE TRUE eu de 0 FALSE 5000 300 Added - Fri Jul 31 11:22:47 2015 \n\n*** scanned and enabled at Fri Jul 31 11:27:04 2015. 0.000 0.000 FALSE FALSE FALSE FALSE 0
4 mirror.tn https://mirror.ati.tn/wiki/ FALSE TRUE af tn 0 FALSE 5000 0 2019-05-25 17:59:54.125 +0000 "Added - Sat Sep 19 20:25:51 2015
*** reenabled by mirrorprobe at Sat May 25 17:47:28 2019." 0.000 0.000 FALSE FALSE FALSE FALSE 0
9 fau.de https://ftp.fau.de/kiwix/ ftp://ftp.fau.de/kiwix/ TRUE TRUE eu de 0 FALSE 5000 636 2023-01-27 15:40:24.841 +0000 Added - Thu Mar 12 17:53:10 2015 Friedrich-Alexander-Universität https://www.fau.de/ 0.000 0.000 FALSE FALSE FALSE FALSE 0
18 mirror-sites-ca.mblibrary.info https://mirror-sites-ca.mblibrary.info/mirror-sites/download.kiwix.org/ rsync://mirror-sites-ca.mblibrary.info/download.kiwix.org/ TRUE TRUE na ca 0 FALSE 100 0 Added - Wed Aug 16 09:43:38 2023 MB Group https://www.mbgroup.global/ 0.000 0.000 FALSE FALSE FALSE FALSE 0
17 mirror-sites-fr.mblibrary.info https://mirror-sites-fr.mblibrary.info/mirror-sites/download.kiwix.org/ rsync://mirror-sites-fr.mblibrary.info/download.kiwix.org/ TRUE TRUE eu fr 0 FALSE 100 0 Added - Wed Aug 16 09:36:48 2023 MB Group https://www.mbgroup.global/ 0.000 0.000 FALSE FALSE FALSE FALSE 0
10 isoc.org.il https://mirror.isoc.org.il/pub/kiwix/ TRUE TRUE EU IL 0 FALSE 100 0 2023-01-27 15:40:28.352 +0000 Added - Sun Jul 17 14:22:46 2011 Israel Internet Association https://isoc.org.il 31.500 34.750 TRUE FALSE FALSE FALSE 0
12 nluug.nl https://ftp.nluug.nl/pub/kiwix/ ftp://ftp.nluug.nl/pub/kiwix/ rsync://ftp.nluug.nl/kiwix/ TRUE TRUE eu nl 0 FALSE 3000 494 2023-01-27 15:40:27.225 +0000 Added - Tue Feb 28 14:18:32 2017 NLUUG https://nluug.nl 52.094 5.119 FALSE FALSE FALSE FALSE 0
2 mirror.accum.se https://saimei.ftp.acc.umu.se/mirror/kiwix.org/ ftp://mirror.accum.se/mirror/kiwix.org/ rsync://mirror.accum.se/mirror/kiwix.org/ TRUE TRUE eu se 0 FALSE 3000 83 2023-01-27 15:40:24.117 +0000 Added - Wed Jun 7 11:34:05 2017 Academic Computer Club in Umeå https://www.accum.se 59.329 18.069 FALSE FALSE FALSE FALSE 0
16 laotzu-mirror.accum.se https://laotzu.ftp.acc.umu.se/mirror/kiwix.org/ ftp://mirror.accum.se/mirror/kiwix.org/ rsync://mirror.accum.se/mirror/kiwix.org/ TRUE TRUE eu se 0 FALSE 3000 0 Added - Tue Mar 21 11:01:30 2023 Academic Computer Club in Umeå https://www.accum.se 59.329 18.069 FALSE FALSE FALSE FALSE 0
14 your.org https://ftpmirror.your.org/pub/kiwix/ ftp://ftpmirror.your.org/pub/kiwix/ rsync://ftpmirror.your.org/pub/kiwix/ TRUE TRUE na us 0 FALSE 5000 0 2023-01-27 15:40:31.488 +0000 "Added - Wed Oct 9 10:52:56 2013
*** scanned and enabled at Sat Nov 9 16:45:16 2013." Your.org https://your.org 0.000 0.000 FALSE FALSE FALSE FALSE 0
1 dotsrc.org https://mirrors.dotsrc.org/kiwix/ ftp://mirrors.dotsrc.org/kiwix/ rsync://mirrors.dotsrc.org/kiwix/ TRUE TRUE eu dk 0 FALSE 5000 0 2023-01-27 15:40:21.219 +0000 Added - Wed Mar 1 11:22:34 2017 dotsrc https://dotsrc.org 55.676 12.568 FALSE FALSE FALSE FALSE 0
15 md.mirrors.hacktegic.com https://md.mirrors.hacktegic.com/kiwix-md/ rsync://md.mirrors.hacktegic.com/kiwix-md/ TRUE TRUE eu md 0 FALSE 3000 0 Added - Mon Feb 13 09:41:46 2023 md.mirrors.hacktegic.com https://md.mirrors.hacktegic.com/ 46.940 28.916 FALSE FALSE FALSE FALSE 0
11 wikimedia https://dumps.wikimedia.org/kiwix/ TRUE TRUE na us 0 FALSE 3000 0 2023-01-27 15:40:26.346 +0000 Added - Sun May 15 09:58:02 2011 Wikimedia Foundation https://wikimedia.org 37.770 -122.393 FALSE FALSE FALSE FALSE 0
19 mirror-sites-in.mblibrary.info https://mirror-sites-in.mblibrary.info/mirror-sites/download.kiwix.org/ rsync://mirror-sites-in.mblibrary.info/download.kiwix.org/ TRUE TRUE as in 0 FALSE 100 0 Added - Wed Aug 16 09:49:41 2023 MB Group https://www.mbgroup.global/ 0.000 0.000 FALSE FALSE FALSE FALSE 0
13 mirrorservice.org https://www.mirrorservice.org/sites/download.kiwix.org/ ftp://ftp.mirrorservice.org/sites/download.kiwix.org/ rsync://ftp.mirrorservice.org/download.kiwix.org/ TRUE TRUE eu gb 0 FALSE 5000 0 2023-01-27 15:40:22.308 +0000 "Added - Sat Nov 9 16:34:19 2013
*** scanned and enabled at Sat Nov 9 16:41:26 2013.
*** scanned and enabled at Sat Nov 9 16:42:27 2013." University of Kent https://www.kent.ac.uk/computing 0.000 0.000 FALSE FALSE FALSE FALSE 0

mirrobrain.csv

@kelson42, from our recent call, I suppose the *_only params might not be properly set.

elfkuzco commented 5 months ago

Taking into account the location of users, location of mirrors, individual content of the mirrors, capabilities of each mirror (how much load they can take) and the online status of the mirrors, the logic for the selection of a particular mirror is done with the following considerations:

According to a section of the video conference, MirrorBrain works in the background. As such, there is no direct way to override the selection decision. However, collecting download statistics can help the content creators define fallback mirrors for countries with low internet connectivity as there is a bit of randomization done even after taking into account the closeness and strength of a mirror. This is highlighted in this section of the video conference where the closest mirror might not always be the best.

kelson42 commented 5 months ago

@kelson42, from our recent call, I suppose the *_only params might not be properly set.

Indeed probably, even if the priority was historicaly given on the overall robustness (to the opposite to speed). Still glas to see that these option are like I remembered them :)

rgaudin commented 5 months ago

Thank you very much @elfkuzco ; that's very helpful.

I'd like to share that our mirrorbrain image is at https://github.com/kiwix/container-images/tree/main/mirrorbrain. This tells me that we are not using mod_asn which could be an enhancement to our setup.

I am closing this issue as I think we have enough information at this stage. We now need data to help us better configure it :)