brl fetch fedora finds incorrect/nonexistent mirrors

[ghost]

After running brl fetch, everytime it finds a nonexistent mirror "=" and tries to download primary.xml.gz from there. (please ignore my themed terminal)

[mu150]

Try using the --mirror flag. Mirrors can be found at https://admin.fedoraproject.org/mirrormanager. Example: sudo brl fetch fedora --mirror=http://ftp.swin.edu.au/fedora/ I'm also getting errors fetching Fedora but I'm using an unsupported distro so I don't know if the default mirror is bad.

[paradigm]

Try using the --mirror flag. Mirrors can be found at https://admin.fedoraproject.org/mirrormanager. Example: sudo brl fetch fedora --mirror=http://ftp.swin.edu.au/fedora/

:+1:

I'm also getting errors fetching Fedora but I'm using an unsupported distro so I don't know if the default mirror is bad.

brl fetch is mostly independent of the rest of the environment [0]. Having a stratum of an unsupported distro shouldn't break anything.

I can't reproduce any issues here; brl fetch fedora works for me.

Under-the-hood brl fetch gets its Fedora mirror list via:

wget -O- "https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-${target_release}&arch=${distro_arch}" 2>/dev/null |
    sed -e 's,/$,,' |
    sed 's,/[^/]*/[^/]*/[^/]*/[^/]*/[^/]*$,,'

Where ${target_release} is something like 32 and ${distro_arch} is Fedora's term for the instruction set architecture, e.g. x86_64.

Can you both try that out and see if it's also failing for you? That link does some magic to generate a region appropriate list; the magic might break down in your region. If so, I can look into implementing some fallback logic.

[0] Most of the things it does get from the environment are obvious, like networking; if you don't have an internet connection, it obviously wont' work. The only non-obvious exception I can recall is that it uses the calling stratum's certs; if you get a cert error, try installing something like a ca-certificates package in at least one stratum and using sudo strat <stratum> brl-fetch ... to use that stratum's certs.

[bobbbay]

You know, my stupid curiosity is getting the better of me. I can't seem to understand why in the world this isn't consistent. I mean, the only reason I can think of is because of geographical locations, which are used to base mirrorlists on. Sorry if I'm not thinking well, kinda tired at the moment 😅

[mu150]

The distro I'm using (postmarketos) acts unusually with bedrock and it sometimes fails to fetch for any distro. I think it would be better for me to make a separate issue for it. Hopefully @the-cobalt can provide feedback.

[ghost]

Try using the --mirror flag. Mirrors can be found at https://admin.fedoraproject.org/mirrormanager. Example: sudo brl fetch fedora --mirror=http://ftp.swin.edu.au/fedora/

I am aware of using the --mirror flag, but I made this issue because it feels like it should work without manually finding a good mirror.

That link does some magic to generate a region appropriate list; the magic might break down in your region. If so, I can look into implementing some fallback logic.

When I go to the website with release 32 and x86_64 arch, this is the page:

# repo = fedora-32 arch = x86_64 country = US country = CR country = CA 
https://mirrors.rit.edu/fedora/fedora/linux/releases/32/Everything/x86_64/os/
https://mirror.arizona.edu/fedora/linux/releases/32/Everything/x86_64/os/
http://mirrors.syringanetworks.net/fedora/linux/releases/32/Everything/x86_64/os/
(there are more mirrors after this)

I think the sed commands doesn't handle the first line with = signs correctly, and therefore makes brl-fetch think that = is a mirror (or multiple mirrors due to there being 5 = signs in the first line).

[paradigm]

You're right it should filter the comment line out there and isn't. That's something I'll look into fixing.

After gathering a list of potential mirrors, brl fetch tries to find the fastest one. At the time it should figure out (the hard way) that the commented lines aren't valid mirrors. It's a problem here, but I don't think it's the problem. When I go to https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-32&arch=x86_64 I see a commented line up top as well, but I don't run into the same issue.

The mirrorlist page content you've provided looks fine to me. My theory that it might be generating a bad page for you didn't pan out.

The only idea I have remaining is to gather debug and step through it. As root, open /bedrock/libexec/brl-fetch with root permissions. Find the line with step "Determining name" around line 1168 and add set -x right before it. This will make brl fetch dump (verbose) debug information from that point onward next time it's run. With that in place, run

brl fetch fedora 2>&1 | tee /tmp/log

and provide the contents of /tmp/log (e.g. gist).

[ghost]

Here's the full version

+ cat /bedrock/strata/fedora/brl-fetch-mirror
+ mirrors='=
https://mirrors.xmission.com/fedora/linux
http://fedora.mirror.iweb.com/linux
(other mirrors)
http://mirrors.kernel.org/fedora'

is where it finds this nonexistent mirror, after it fast filters, so maybe it has something to do with netselect? (I have no idea how fast filtering works)

[paradigm]

Running the debug on my system, I find my system is feeding the full mirror list from fedoraproject.org/mirrorlist including the comment line into netselect, but my netselect output does not include the = line. I have no idea why we'd be different there. Debugging further into why we're different here remotely may be a pain. While I'm curious to know, the best ROI solution here might just be to ensure we're feeding good values into netselect. That is, take care of the comment line you correctly identified as problematic.

Remove the set -x we placed earlier. Once that's gone, open /bedrock/share/brl-fetch/distros/fedora with root permissions. Find the list_mirrors() function and change

    wget -O- "https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-${target_release}&arch=${distro_arch}" 2>/dev/null |
        sed -e 's,/$,,' |
        sed 's,/[^/]*/[^/]*/[^/]*/[^/]*/[^/]*$,,'

to

    wget -O- "https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-${target_release}&arch=${distro_arch}" 2>/dev/null |
        grep '^http' |
        sed -e 's,/$,,' |
        sed 's,/[^/]*/[^/]*/[^/]*/[^/]*/[^/]*$,,'

adding the grep line. Then try it again. If that resolves the issue for you, I'll ensure the next release includes it. When I get the time, I'll also look into making things more robust by having the code that consumes list_mirrors() output validate things and drop bad values before sending them off to netselect. This way if the same issue exists elsewhere it should be handled transparently.

[ghost]

It works now. Definitely a good idea to add it to the next release. Thanks!

[paradigm]

Thanks for reporting it, being patient with my response time, and working with me to debug it!

[paradigm]

Should be fixed in 0.7.18beta5 that I just pushed. If you have the time, please try it out and confirm for me that things now act as you expect.

[ghost]

Works perfectly! Thanks!