"brl fetch fedora" fails to find mirrors or fetch "primary.xml.gz"

[fogti]

brl fetch -r 30 fedora fails with "Unable to automatically find a valid mirror".

[ 1/18 (  5%)}] Determining name
* Using fedora
[ 2/18 ( 11%)}] Determining CPU architecture
* Using x86_64
[ 3/18 ( 16%)}] Determining release
* Using 30
[ 4/18 ( 22%)}] Determining mirror
* Checking bedrock.conf [brl-fetch-mirror] items
[\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\] 100%
* Fast filtering 151 to top 10
[\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\] 100%
Fast filter failed, falling back to slow method against all 151 mirrors.
This is known to fail within certain virtualization environments which do not support ICMP.
If this seems unacceptably slow, ctrl-c out, brl remove the stratum, then look up a good mirror for the distro and brl fetch again, this time manually providing mirror with -m
* Finding fastest mirror from 151
[\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\] 100%
ERROR: Unable to automatically find a valid mirror.  Manually specify mirror with `--mirror`.
ERROR: Unexpected error occurred.

brl fetch -r 30 -m http://fedora.tu-chemnitz.de/pub/linux/fedora/linux fedora fails with "HTTP/1.1 404 Not Found".

[ 1/18 (  5%)}] Determining name
* Using fedora
[ 2/18 ( 11%)}] Determining CPU architecture
* Using x86_64
[ 3/18 ( 16%)}] Determining release
* Using 30
[ 4/18 ( 22%)}] Determining mirror
* Using http://fedora.tu-chemnitz.de/pub/linux/fedora/linux
[ 5/18 ( 27%)}] Making bootstrap directory structure
[ 6/18 ( 33%)}] Downloading package information database
Looking for file matching: primary.xml.gz
at: http://fedora.tu-chemnitz.de/pub/linux/fedora/linux/releases/30/Everything/x86_64/os/repodata/
found: /pub/linux/fedora/linux/releases/30/Everything/x86_64/os/repodata/54ed9948b74b314b877fa90468e7c5309a0f9428c35b938b82fa7b356a799c94-primary.xml.gz
* use http://fedora.tu-chemnitz.de/pub/linux/fedora/linux/releases/30/Everything/x86_64/os/repodata///pub/linux/fedora/linux/releases/30/Everything/x86_64/os/repodata/54ed9948b74b314b877fa90468e7c5309a0f9428c35b938b82fa7b356a799c94-primary.xml.gz
Connecting to fedora.tu-chemnitz.de (...)
Connecting to ftp.tu-chemnitz.de (...)
wget: server returned error: HTTP/1.1 404 Not Found
ERROR: Unexpected error occurred.

brl-fetch tries to fetch primary.xml.gz from "http://fedora.tu-chemnitz.de/pub/linux/fedora/linux/releases/30/Everything/x86_64/os/repodata///pub/linux/fedora/linux/releases/30/Everything/x86_64/os/repodata/54ed9948b74b314b877fa90468e7c5309a0f9428c35b938b82fa7b356a799c94-primary.xml.gz", which is obviously wrong (path prefix is appended twice in find_link), it should use "http://fedora.tu-chemnitz.de/pub/linux/fedora/linux/releases/30/Everything/x86_64/os/repodata/54ed9948b74b314b877fa90468e7c5309a0f9428c35b938b82fa7b356a799c94-primary.xml.gz" instead,

but I don't know how to fix find_link (in brl-fetch), without breaking other fetch scripts.

[paradigm]

First issue appears to be because path brl fetch uses to sanity check a mirror is no longer available. I try to find one that has a consistent file path and is a good balance of not being so large as to be expensive to download, but not be so small as to have other factors overshadow the download time. I found another we can use, at least for the time being:

Open /bedrock/share/brl-fetch/distros/fedora and change

speed_test_url() {
    suffix="releases/${target_release:-}/Everything/${distro_arch:-}/os"
    echo "${suffix}/LICENSE"
}

to

speed_test_url() {
    echo "releases/${target_release:-}/Everything/${distro_arch:-}/os/repodata/repomd.xml"
}

The second issue appears to be because all the mirrors I tested against link just the file name, implicitly relative to their containing directory in the URL. However, the one you've chosen is relative to the domain, which I apparently didn't test against.

For example,

$ curl 'https://ewr.edge.kernel.org/fedora-buffet/fedora/linux/releases/30/Everything/x86_64/os/repodata/' 2>/dev/null | sed 's/^.*href//' | awk -F'"' '/primary.xml.gz/ {print$2}'
54ed9948b74b314b877fa90468e7c5309a0f9428c35b938b82fa7b356a799c94-primary.xml.gz
$ curl 'http://ftp.tu-chemnitz.de/pub/linux/fedora/linux/releases/30/Everything/x86_64/os/repodata/' 2>/dev/null | sed 's/^.*href//' | awk -F'"' '/primary.xml.gz/ {print$2}'
/pub/linux/fedora/linux/releases/30/Everything/x86_64/os/repodata/54ed9948b74b314b877fa90468e7c5309a0f9428c35b938b82fa7b356a799c94-primary.xml.gz

Note the kernel.org output just has the file name, while tu-chemnitz.de has containing directories.

The lazy solution here is to assume any path with a / is equivalent to the URL and strip everything up to the last /. We can do this by opening up /bedrock/libexec/brl-fetch and changing

    path="$(wget -O- "${index}" 2>/dev/null | list_links | grep -- "${regex}" | head -n1)"

to

    path="$(wget -O- "${index}" 2>/dev/null | list_links | grep -- "${regex}" | head -n1 | sed 's,^.*/,,')"

A possible alternative would be to detect the / and follow the link properly.

Feel free to make these changes locally to fix things on your system for the time being. I'll queue them up with the other stuff planned for 0.7.7. Once 0.7.7 releases, it is likely to overwrite your changes should I have found better solutions during testing.