search

manjaro / pacman-mirrors

This repo has been archived. Our code is now hosted at
https://gitlab.manjaro.org
GNU General Public License v3.0
25 stars 16 forks source link

Multiple entries of same server in custom-mirrors.json #116

Closed petsam closed 6 years ago

petsam commented 6 years ago

In interactive mode, mirrors with more than 1 protocol are entered multiple times in custom-mirrors.json In the window that presents selectable mirrors the server are listed for as many protocols they support If a user selects the same server in more than one of its protocols pacman-mirrors writes them in custom-mirrors.json as many times in exactly the same entries. Thus they are polled those many times whenever a poll is needed. When an -f command or similar is executed, it takes these several times the same mirror from custom-mirrors.json to poll. So it assumes as they are different mirrors and at the end it sorts and writes them all in the mirrorlist, depending of course the response time. It is a fact that the same server responds in random speeds (milliseconds) at polls, so it is not ordered in the mirrorlist in sequence.

sudo pacman-mirrors -aS testing

   ... ...
   0.589 Greece         : https://ftp.cc.uoc.gr/mirrors/linux/manjaro/
   0.347 Greece         : http://ftp.cc.uoc.gr/mirrors/linux/manjaro/
   1.757 Greece         : ftp://ftp.cc.uoc.gr/mirrors/linux/manjaro/
   1.394 Greece         : https://ftp.cc.uoc.gr/mirrors/linux/manjaro/
   0.283 Greece         : http://ftp.cc.uoc.gr/mirrors/linux/manjaro/
   1.508 Greece         : ftp://ftp.cc.uoc.gr/mirrors/linux/manjaro/
   0.362 Greece         : https://ftp.cc.uoc.gr/mirrors/linux/manjaro/
   0.244 Greece         : http://ftp.cc.uoc.gr/mirrors/linux/manjaro/
   ..... Greece         : ftp://ftp.cc.uoc.gr/mirrors/linux/manjaro/
   ... ...
   0.354 Netherlands    : http://ftp.snt.utwente.nl/pub/linux/manjaro/
   1.347 Netherlands    : ftp://ftp.snt.utwente.nl/pub/linux/manjaro/
   ..... Netherlands    : http://ftp.snt.utwente.nl/pub/linux/manjaro/
   1.414 Netherlands    : ftp://ftp.snt.utwente.nl/pub/linux/manjaro/
   1.111 Netherlands    : https://ftp.nluug.nl/pub/os/Linux/distr/manjaro/
   1.326 Netherlands    : ftp://ftp.nluug.nl/pub/os/Linux/distr/manjaro/
   0.684 Netherlands    : https://ftp.nluug.nl/pub/os/Linux/distr/manjaro/
   1.280 Netherlands    : ftp://ftp.nluug.nl/pub/os/Linux/distr/manjaro/
   ... ...
   0.801 Sweden         : https://ftp.lysator.liu.se/pub/manjaro/
   1.142 Sweden         : ftp://ftp.lysator.liu.se/pub/manjaro/
   0.680 Sweden         : https://ftp.lysator.liu.se/pub/manjaro/
   ..... Sweden         : ftp://ftp.lysator.liu.se/pub/manjaro/
   0.363 Turkey         : http://ftp.linux.org.tr/manjaro/
   0.946 Turkey         : ftp://ftp.linux.org.tr/manjaro/
   0.411 Turkey         : http://ftp.linux.org.tr/manjaro/
   0.956 Turkey         : ftp://ftp.linux.org.tr/manjaro/
   ... ...

The Greek mirror as an example

    {
        "country": "Greece",
        "protocols": [
            "https",
            "http",
            "ftp"
        ],
        "url": "ftp://ftp.cc.uoc.gr/mirrors/linux/manjaro/"
    },
    {
        "country": "Greece",
        "protocols": [
            "https",
            "http",
            "ftp"
        ],
        "url": "ftp://ftp.cc.uoc.gr/mirrors/linux/manjaro/"
    },
    {
        "country": "Greece",
        "protocols": [
            "https",
            "http",
            "ftp"
        ],
        "url": "ftp://ftp.cc.uoc.gr/mirrors/linux/manjaro/"
    },

    {
        "branches": [
            1,
            0,
            0
        ],
        "country": "Greece",
        "last_sync": "10:45",
        "protocols": [
            "ftp",
            "http",
            "https"
        ],
        "url": "https://ftp.cc.uoc.gr/mirrors/linux/manjaro/"
    },

 {"country": "Greece", "url": "https://ftp.cc.uoc.gr/mirrors/linux/manjaro/", "protocols": ["ftp", "http", "https"]},
fhdk commented 6 years ago

If a mirror for some unknown reason is registered several times in either share/pacman-mirrors/mirrors.json or lib/pacman-mirrors/status.json they will occur several times if included by country in custom-mirrors.json.

Pacman-mirrors is not responsible for the content of status.json and mirrors.json and does not question the validity of the fetched datafiles.

If such duplication occurs the right place to change it is by opening an issue at

manjaro/manjaro-web-repo/issues

As of this comment I have created a custom-mirrors.json by supplying for countries with -c

https://pastebin.com/uBBF2QPW

If you use -i pacman-mirrors makes no assumptions on which protocols you prefer and will probe and list all available protocols for you to decide which ones you prefer.

~ >>> sudo pacman-mirrors -f -i                                                            
[sudo] password for fh: 
.: INFO Downloading mirrors from repo.manjaro.org
.: INFO Using custom mirror file
.: INFO Querying mirrors - This may take some time
   0.419 Greece         : https://ftp.cc.uoc.gr/mirrors/linux/manjaro/
   0.174 Greece         : http://ftp.cc.uoc.gr/mirrors/linux/manjaro/
   0.803 Greece         : ftp://ftp.cc.uoc.gr/mirrors/linux/manjaro/
   0.112 Netherlands    : https://ftp.nluug.nl/pub/os/Linux/distr/manjaro/
   0.168 Netherlands    : ftp://ftp.nluug.nl/pub/os/Linux/distr/manjaro/
   0.166 Netherlands    : http://ftp.snt.utwente.nl/pub/linux/manjaro/
   0.208 Netherlands    : ftp://ftp.snt.utwente.nl/pub/linux/manjaro/
   0.103 Netherlands    : https://nl.mirror.babylon.network/manjaro/
   0.158 Netherlands    : https://manjaro.mirror.wearetriple.com/
   0.128 Netherlands    : https://mirror.koddos.net/manjaro/
   0.917 Sweden         : https://ftp.lysator.liu.se/pub/manjaro/
   0.230 Sweden         : ftp://ftp.lysator.liu.se/pub/manjaro/
   0.132 Sweden         : https://mirror.zetup.net/manjaro/
   0.225 Turkey         : http://ftp.linux.org.tr/manjaro/
   0.615 Turkey         : ftp://ftp.linux.org.tr/manjaro/

This is not a bug - it is a feature - since pacman-mirrors can make no assumptions on your behalf.

petsam commented 6 years ago

Sorry to continue trying... As you said:

If a mirror for some unknown reason is registered several times in either share/pacman-mirrors/mirrors.json or lib/pacman-mirrors/status.json

Per my description:

In /var/lib/pacman-mirrors/status.json

and

In /usr/share/pacman-mirrors/mirrors.json

this is obvious is not the case. I was not describing a random use of -i but the one that is used as only parameter and uses the Gui to show the available mirrors for selection. In that window the protocols are not combined as in the Manjaro repo JSONs, which are correct. In a 3 protocol server, the user is given the impression that there is a choice of 3 mirrors for the same server that he may or may not select individually. If he chooses 2 or 3, then the script creates that many identical entries and parses them as different which by the JSONs they are not.

Thanks again for your time and effort.

fhdk commented 6 years ago

If I understand you correct

Don't you agree that the user should make the decision on which prototols to use?

If the ftp version of the mirror is less used and therefore more reliable it should be the user's choice not pacman-mirrors.

Which is why the --interactive mode is really interactive.

petsam commented 6 years ago

I am not native English speaking, so I am used to being misunderstood. I thing you did not understand my point of discussion. It is the code, not the reason. For a discussion about the logic Manjaro uses to update the mirrors I tried to initiate a discussion, but it was not honored enough. If you have something to say about that, it would be better for all to do it in the forum.

About the code, I describe clearly enough what the issue is and yet you have just closed the issue, without waiting for my comment on your assumption for the correct (or not) interpretation of my description. I am sorry you have made my struggle to give useful feedback to the Manjaro team (and not one contributor in personal manner) look like a personal vendetta. I am really sorry. I don't have the slightest intention nor the character for that. I love Open Source Software, so I respect all code contributors. I learn't that open source means "we are all together and each one helps the other to improve". I tried to honor that, which I have received from others and try to improve myself in humility. I could have sent you a private message for all the above, but I thought hard if I should, thinking of you might be really angry. I think, if everyone hesitates to tell you something probably irritating but truthful and innocent then you can never advance yourself. So try to gain from this "rudeness" of mine, please. And be sure I have no problem with you, really. Peace!

fhdk commented 6 years ago

@petsam I closed the issue - not because I didn't want to hear your opinion but - because I see no issue.

You are entitled to your opinion that it is confusing, that all protocols is presented and available for selection.

As the purpose of the --interactive mode is to allow full control to the user I like to think the users are smart enough to know that one mirror offers 3 different means of fetching packages and they can select the one that suits their circumstances best.

If they choose to select all 3 - who am I to say that is wrong?

I am sorry you have made my struggle to give useful feedback to the Manjaro team (and not one contributor in personal manner) look like a personal vendetta.

I think you are the only one to look at it that way - I certainly don't :vulcan_salute:

petsam commented 6 years ago

all protocols is presented and available for selection

This is not the problem. The problem is that when a user wants more than one protocol for the same server, he cannot have it if he checks them all (the different protocol entries) because there is no significance for pacman-mirrors which when it writes them in custom config, 3 identical entries are created with no mention on which protocol each entry represents and when it reads them to (re)create the mirrorlist cannot distinguish they are the same server with different protocol preference, hence uses them repeatedly with the same effect, for no benefit. As far I understood from the man pages and reading the code, the only way to set some preference to protocols is in pacman-mirrors.conf and it is set for all mirrors, cannot be set per server. This is the info text

## Define protocols and priority
##   separated by comma 'https,http' or 'http,https'
## ATM available protocols are: http, https, ftp
## Not specifying a protocol will ban the protocol from being used
## If a mirror has more than one protocol defined only the first is written to the mirrorlist
## Empty means all in reversed alphabetic order
# Protocols =

If they choose to select all 3 - who am I to say that is wrong?

The user is always responsible of course. But giving the option to the user to select three entries for one server you give him the impression that he has a choise for the protocol, which is not true, as it is stated in pacman-mirrors.conf clearly. Thus users are misguided from the GUI behavior which is designed as it is now. And other info say differently. ?? Nevertheless, if what I am saying is wrong, maybe the information given in --help, man, wiki, .conf, the forum topics, may need some house cleaning. It is too overwhelming. I am very confident with my understanding of these technical stuff, from a user's perspective. And my care is for the users, not myself. I have already setup custom mirrorlist, frozen. It's the Arch way, which in this case of the mirrors makes perfect sense. Mirrors shouldn't change frequently, just in case of trouble or urgent need of a new update and lack of patience.

And for the closing... do you know that all the effort of pacman-mirrors to create a custom mirrorlist for the inexperienced user is going to the "trash"? The inexperienced user would usually prefer Pamac for updates for convenience and friendliness. Pamac does not use the mirrorlist created by pacman-mirrors in /etc/pacman.d/mirrorlist as it states always. Pamac (update daemon and normal GUI) creates an on-the-fly mirrorlist with pacman-mirrors, no matter the existing mirrorlist. So, for what all this trouble??

And again, I may be wrong, actually, I really hope so.

fhdk commented 6 years ago

The problem is that when a user wants more than one protocol for the same server

giving the option to the user to select three entries for one server you give him the impression that he has a choise for the protocol

The documentation is very clear about the intention of --interactive and I quote:

This is a function designed to leave full control over countries, mirrors and protocols to the user.

Indeed the user has a choice, however if the user chooses several protocols, then the actual used protocols will vary depending on the response time because all will all be probed and only the fastest is written to the mirrorlist. Which is exactly what the documentation tells. I fail to see how this is a problem that needs addressing.

I have already setup custom mirrorlist, frozen

So do I. I have only one mirror - that's all.

do you know that all the effort of pacman-mirrors to create a custom mirrorlist for the inexperienced user is going to the "trash"

For Pamac - it does nothing regarding creating a mirrorlist or settings. Pamac uses pacman-mirrors. You can check by experimenting with different settings of your pacman-mirrors and then use pamac to refresh mirrors - look at the output from pamac in the details window. Pamac does not create on-the-fly mirrorlist - it uses libalpm for installation which is a pacman library with in turn depends on the pacman.conf file.

So the pacman-mirrors effort is not wasted - it is the defacto tool for mirrorlist creation in manjaro.