Feature request -- Specify country/country list for mirror auto-selection

[cmsigler]

Hi,

I feel this falls into a general category of "Improve 'auto' mirror selection" but that seems too general for a bug report IMHO. Instead I would like to propose a feature request instead. As always I hope this helps :)

My feature request: Please consider a variable to restrict selected mirrors to a single country or a list of countries.

My mirror auto-selection seems not to have worked too well :( Perhaps I can help because I'm a US user(?). I find I have mirrors from Sweden and Australia in my auto-selection list, as well as United States and Canada (Canada should be fine, of course). (FYI, I have 5 US, 2 Canada, and 1 each Australia and Sweden. Also, of the 5 US mirrors only one of flexo's were in my old mirrorlist selected via rankmirrors from exclusively US mirrors :( And I have learned how to use rankmirrors to manually tune my mirrorlist very efficiently, again, FWIW....)

If there would be a config option in flexo.toml to restrict countries from which mirrors are chosen, this could be improved, I propose. Just my 2c which I hope you find useful and something you wish to implement :)

Thank you!

Clemmitt

Edit: Turns out I have only one feature request. I just discovered that the auto-selected mirrorlist can be regenerated simply by restarting flexo :)

[cmsigler]

Hi,

One more observation now that I'm playing with this a bit more. For some reason the auto-selection of mirrors always seems to select the same mirrors. However, none of these mirrors is a "fast" mirror according to rankmirrors, and rankmirrors has served me very well.

The auto-selected list just looks "strange" based on my manually managing my mirrorlist for a very long time in order to saturate my DSL line. These seem to be slow compared to my manually managed list.

Obviously the mirrors chosen come from the latest archlinux.org mirrors status JSON. I assume when I run rankmirrors it does the same thing. The only difference is I restrict rankmirrors to those from the US.

Again, I really appreciate flexo because I really need a pacman pkg proxy cache since I now find myself with four Arch boxes at home (I'm a masochist ;). I hope my comments will help you. Thank you again very much!

Clemmitt

[nroi]

Thanks for reporting this. I would like to prioritize the auto-selection mechanism before implementing workarounds to circumvent the less-than-ideal auto-selection mechanism.

Could you paste your /var/cache/flexo/state/latency_test_results.json? This might help me troubleshoot this issue further.

[cmsigler]

Hi,

Well I wish I could send you a copy of the old latency_test_results.json, but when I restarted flexo after finding mirrors from Sweden and Australia it was overwritten :(

I haven't read your program. Is it supposed to have "adaptive logic" that learns over time? (I'm guessing it isn't....)

I can't explain it, but now flexo is choosing much closer mirrors all in the US. Why it had that glitch when I was first trying it out is beyond me.

Just FYI, here is the current latency_test_results.json:

---

{ "timestamp": "2020-12-07T18:03:31.023427394Z", "download_providers": [ { "uri": "https://arch.mirror.constant.com/", "mirror_results": { "total_time": { "secs": 0, "nanos": 222952000 }, "namelookup_duration": { "secs": 0, "nanos": 3235000 }, "connect_duration": { "secs": 0, "nanos": 44685000 }, "pretransfer_time": { "secs": 0, "nanos": 151003000 }, "starttransfer_time": { "secs": 0, "nanos": 222889000 } }, "country": "United States" }, { "uri": "https://mirror.dal10.us.leaseweb.net/archlinux/", "mirror_results": { "total_time": { "secs": 0, "nanos": 294726000 }, "namelookup_duration": { "secs": 0, "nanos": 2232000 }, "connect_duration": { "secs": 0, "nanos": 64915000 }, "pretransfer_time": { "secs": 0, "nanos": 228982000 }, "starttransfer_time": { "secs": 0, "nanos": 294661000 } }, "country": "United States" }, { "uri": "https://arlm.tyzoid.com/", "mirror_results": { "total_time": { "secs": 0, "nanos": 331767000 }, "namelookup_duration": { "secs": 0, "nanos": 2473000 }, "connect_duration": { "secs": 0, "nanos": 86599000 }, "pretransfer_time": { "secs": 0, "nanos": 232167000 }, "starttransfer_time": { "secs": 0, "nanos": 331717000 } }, "country": "United States" }, { "uri": "https://mirror.wdc1.us.leaseweb.net/archlinux/", "mirror_results": { "total_time": { "secs": 0, "nanos": 286177000 }, "namelookup_duration": { "secs": 0, "nanos": 3678000 }, "connect_duration": { "secs": 0, "nanos": 87520000 }, "pretransfer_time": { "secs": 0, "nanos": 249499000 }, "starttransfer_time": { "secs": 0, "nanos": 286130000 } }, "country": "United States" }, { "uri": "https://arch.hu.fo/archlinux/", "mirror_results": { "total_time": { "secs": 0, "nanos": 309015000 }, "namelookup_duration": { "secs": 0, "nanos": 2321000 }, "connect_duration": { "secs": 0, "nanos": 93721000 }, "pretransfer_time": { "secs": 0, "nanos": 208679000 }, "starttransfer_time": { "secs": 0, "nanos": 308965000 } }, "country": "United States" } ] }

---

It would probably help to know I am (physically and) network hop-wise located close to the Washington, DC/metro area, so the mirror.wdc1.us.leaseweb.net mirror is a good one and is also retrieved when I use rankmirrors.

One other peculiarity of my network connection -- My ISP is a dinosaur and refuses to support IPv6. So I use a Hurricane Electric 6-in-4 tunnel with the other endpoint in the DC area. This means my IPv4 and IPv6 routing are asymmetric. FWIW....

Please let me know how else I can help :) Thank you again!

Clemmitt

Edit: Correct stupid wiki editing mode mistake; mention IPv6 tunnel config

[nroi]

Thanks for the detailed info, I'm going to collect everything related to the auto-selection issue in #29. I'll keep this issue open for the feature request of including a list of mirrors.

[nroi]

This has been implemented on version 1.0.9. To make use of this setting, proceed as follows:

• Install the most recent version of Flexo.
• Edit your /etc/flexo/flexo.toml: Add something like allowed_countries = ["US"] (or whatever country makes sense for you) at the bottom of the mirrors_auto section.
• Restart Flexo.
• Wait for 10 seconds or so.

The file /var/cache/flexo/state/latency_test_results.json should then include only mirrors from the US.

[cmsigler]

Hi,

Thank you for your hard work to implement this, and for flexo :)

Clemmitt