Add option to only use the deps repository

[sbernhard]

This change is necessary to bootstrap oracle8. On oracle8, there is no subscription-manager by default in the repositories. Therefore you need to use a self-build sub-man which is part of a repository. The target of this PR is, to use ONLY this deps repositories which stores the sub-man because the Oracle8 base repositories can not be used as they require to login to the "orcale network" first.

[evgeni]

Honestly, I hate adding more options. Especially as it's not obvious to the user why they should use it from the help text.

Do I assume correctly that this is required because a basic OL8 install has OL repos in /etc/yum.repos.d which are unavailable and a dnf install <something> fails due to that?

I'd rather prefer us either instructing dnf not to fail in this case (I think it has a CLI switch for that, so there also should be an API call), or a try/except around dnf_base.fill_sack() (if that's the failing line?)

[evgeni]

Alternatively, drop the option and make it if options.deps_repository_url?

[sbernhard]

Do I assume correctly that this is required because a basic OL8 install has OL repos in /etc/yum.repos.d which are unavailable and a dnf install <something> fails due to that?

Yes. This is exactly what happens. If dnf fails, it does not find the sub-man and therefore its not possible to install sub-man.

I guess, "if options....url?" (your alternative solution) would harm some other scenarios in which deps-repo + default repos are necessary.

Well, there is another solution which doesn't affect the code: dear oracle users, remove / rename the original yum repos before you want to bootstrap them.... :-)

[evgeni]

So I thought about that a bit more and either I'm not understanding the issue correctly, or the change shouldn't fix it...

Let me reiterate:

• You are trying to subscribe an EL8 (more precisely Oracle Linux 8) system
• That system has unreachable repositories (Oracle) configured and enabled
• dnf dislikes the fact that they are unavailable and errors out instead of finding sub man in the provided additional bootstrap repository

Now, your change modifies the code that does "search for sub man", but not the code that does "install subman", so even after the change you should not end up with a working system, as the install step should fail?

[evgeni]

Another question that comes to mind: why are those repos unavailable? Bootstrap is supposed to migrate brown field environments, so I'd expect working systems with working repos, not something broken ;)

(Doesn't mean it shouldn't work, just that the use of bootstrap is questionable)

[sbernhard]

1. Oracle ships the default repo files but you need to be logged in to the Unbreakable Linux Network to use them. If you use them without being logged-in, you will get an error.
2. Oracle doesn't provide the sub-man package. So you need to use another repository containing the sub-man package. e.g. from CentOS

=> therefore, the approach is to use the deps-repository (2.) without running into (1.)

[evgeni]

Right, I got that. But:

1. When we call "install", we don't disable that repository: https://github.com/Katello/katello-client-bootstrap/blob/12d1a0cb9bd62dcb9ef09182416eef3447fb0b0b/bootstrap.py#L287-L297
2. The same issue should also happen on Oracle 7 (at least CentOS7 refuses to search/install when there is an unavailable repo in /etc/yum.repos.d)
3. Is that a freshly installed machine? If so, why is bootstrap even used instead of the real provisioning in Foreman. Again, not saying it shouldn't work this way, just it's suboptimal.

[sbernhard]

1) good question. Maybe its "dnf vs yum"? While installing sub-man it works without an error. The error happend before while verifying if sub-man is already installed or not. 2) maybe again "dnf vs. yum" - maybe the yum doesn't complains not usable repos when verifying if sub-man is already installed 3) well, we have tested that with a "base oracle8" installation as a reference. Of course, we are using Foreman to provision new systems :)

[evgeni]

[root@a1245aa8ea8f /]# cat /etc/centos-release
CentOS Linux release 8.2.2004 (Core) 
[root@a1245aa8ea8f /]# cat /etc/yum.repos.d/fail.repo
[fail]
name=fail
baseurl=http://192.168.43.1:8000/not/existing
gpgcheck=0
[root@a1245aa8ea8f /]# yum install vim
Failed to set locale, defaulting to C.UTF-8
fail                                                                                                                                                                                                   0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'fail':
  - Curl error (7): Couldn't connect to server for http://192.168.43.1:8000/not/existing/repodata/repomd.xml [Failed to connect to 192.168.43.1 port 8000: Connection refused]
Error: Failed to download metadata for repo 'fail': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
[root@a1245aa8ea8f /]# echo $?
1
[root@a1245aa8ea8f /]# dnf install vim
Failed to set locale, defaulting to C.UTF-8
fail                                                                                                                                                                                                   0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'fail':
  - Curl error (7): Couldn't connect to server for http://192.168.43.1:8000/not/existing/repodata/repomd.xml [Failed to connect to 192.168.43.1 port 8000: Connection refused]
Error: Failed to download metadata for repo 'fail': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
[root@a1245aa8ea8f /]# echo $?
1

[evgeni]

So the real issue is the ociregion variable in Oracle's repo files.

Simple test script:

import dnf

def run_dnf():
        dnf_base = dnf.Base()
        dnf_base.conf.read()
        dnf_base.read_all_repos()
        dnf_base.fill_sack()
        pkg_list = dnf_base.sack.query().filter(name='subscription-manager')
        print(pkg_list.installed().run())
        print(pkg_list.available().run())

run_dnf()

This errors out on OL8 with:

Errors during downloading metadata for repository 'ol8_baseos_latest':
  - Curl error (6): Couldn't resolve host name for https://yum$ociregion.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64/repodata/repomd.xml [Could not resolve host: yum$ociregion.oracle.com]
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/dnf/repo.py", line 573, in load
    ret = self._repo.load()
  File "/usr/lib64/python3.6/site-packages/libdnf/repo.py", line 394, in load
    return _repo.Repo_load(self)
RuntimeError: Failed to download metadata for repo 'ol8_baseos_latest': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./test.py", line 13, in <module>
    run_dnf()
  File "./test.py", line 8, in run_dnf
    dnf_base.fill_sack()
  File "/usr/lib/python3.6/site-packages/dnf/base.py", line 392, in fill_sack
    self._add_repo_to_sack(r)
  File "/usr/lib/python3.6/site-packages/dnf/base.py", line 137, in _add_repo_to_sack
    repo.load()
  File "/usr/lib/python3.6/site-packages/dnf/repo.py", line 580, in load
    raise dnf.exceptions.RepoError(str(e))
dnf.exceptions.RepoError: Failed to download metadata for repo 'ol8_baseos_latest': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

Adding a dnf_base.conf.substitutions['ociregion'] = '' to the script above (after dnf_base.conf.read()) makes it work like a charm. (see docs for the config)

Why this variable is not set when you use the python library directly, I don't know. But adding this conditionally on Oracle systems sounds like the correct fix here.

[evgeni]

That variable is loaded from /etc/dnf/vars/ociregion, added in Oracle BZ 30121584 (but I can't find a working link to it)

[sbernhard]

So you think, there no change necessary in bootstrap script because the ociregion variable will be available soon and then the issue is fixed?

[evgeni]

No, dnf seems not to read that variable properly when called from Python, so we need to workaround it (or tell dnf to read, I have no idea how)