search

teemtee / tmt

Test Management Tool
MIT License
80 stars 121 forks source link

Introduce a new `prepare` plugin for common features #1062

Closed psss closed 7 months ago

psss commented 2 years ago

As the first step towards a consistent way to prepare guests for testing a new feature plugin for the prepare step was discussed. The expected config syntax looks like this:

prepare:
    how: feature
    epel: enabled
    crb: disabled

Similarly there should be a L1 key feature which could be used to enabled features for individual tests:

test: ./test.sh
feature:
    epel: enabled
    fips: enabled

For the implementation an ansible role with playbooks covering individual features should be used.

psss commented 2 years ago

For the implementation an ansible role with playbooks covering individual features should be used.

There are some limitations for ansible implementation on rhel (e.g. only some modules available). Double-check this before implementing as we need this to be working reliable across all supported distros.

idorax commented 2 years ago

@psss, I'll look into this issue.

psss commented 2 years ago

@psss, I'll look into this issue.

Great! As the first proof of concept you can start with creating just the new feature prepare plugin (we can keep the L1 functionality for later). The install plugin can serve you as a good template for start.

idorax commented 2 years ago

Before implementing this plugin, let me explain the three abbreviations listed above from my understanding,

@psss, if what I understood on epel, crb and fips is not exactly right, please help to correct me, thanks!

idorax commented 2 years ago

@psss, would you please help to explain what should to be done on guest if epel is enabled? From my understanding, if epel is enabled, we should try to install the package epel-release on guest, right?

psss commented 2 years ago

@psss, if what I understood on epel, crb and fips is not exactly right, please help to correct me, thanks!

Yes, the summaries and links are correct.

@psss, would you please help to explain what should to be done on guest if epel is enabled?

I'd say we should follow the quickstart instructions from the official documentation. They clearly specify which specific steps need to be performed on different rhel/centos versions.

From my understanding, if epel is enabled, we should try to install the package epel-release on guest, right?

Yes, basically making sure that the epel-release is installed on the guest and the repository is enabled.

idorax commented 11 months ago

Hi @psss and @happz, in the process of revising PR #2198 , I noticed that epel and crb are closely related (as command /usr/bin/crb is from package epel-release). So to simply both user interface and code implementation, can we merge them together?

Let me take RHEL 8 for example, if user specifies:

prepare:
    how: feature
    epel: enabled

What the plugin should do:

What do you think?

happz commented 11 months ago

Hi @psss and @happz, in the process of revising PR #2198 , I noticed that epel and crb are closely related (as command /usr/bin/crb is from package epel-release). So to simply both user interface and code implementation, can we merge them together?

That does not sound right to me. The eventual effect, I mean, there might be shared parts of implementation (and there are, classes here and there), and that's fine.

Let me take RHEL 8 for example, if user specifies:

prepare:
    how: feature
    epel: enabled

In this scenario, I would expect EPEL repo to be enabled, with no other changes. I did not ask for CRB to be enabled or disabled, therefore I'd expect CRB - or any other feature - to be left untouched, unmodified.

What the plugin should do:

  • Make sure packages related to epel (i.e. epel-release and epel-release) are installed on the guest 
    $ sudo dnf -y install \
  https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
$ sudo dnf -y install \
  https://dl.fedoraproject.org/pub/epel/epel-next-release-latest-8.noarch.rpm

Sounds correct.

  • Enable repos defined in /etc/yum.repos.d/epel.repo and /etc/yum.repos.d/epel-next.repo on the guest 
    # $ grep -E "\[.*\]" /etc/yum.repos.d/epel.repo
# [epel]
# [epel-debuginfo]
# [epel-source]
# $ grep -E "\[.*\]" /etc/yum.repos.d/epel-next.repo
# [epel-next]
# [epel-next-debuginfo]
# [epel-next-source]
$ sudo dnf config-manager --enable \
 epel epel-debuginfo epel-source \
 epel-next epel-next-debuginfo epel-next-source

Again, makes sense, with epel: enabled, I'd expect EPEL repos to be enabled.

  • Enable the codeready-builder(CRB) or equivalent repo on the guest 
    # $ rpm -ql epel-release epel-next-release | grep crb
# /usr/bin/crb
# $ rpm -qf /usr/bin/crb
# epel-release-8-19.el8.noarch
$ /usr/bin/crb enable

Here I disagree - I did not ask for this.

idorax commented 11 months ago

Hi @happz, thanks very much for clarification! BTW, if user specifies:

prepare:
   how: feature
   crb: enabled

The plugin should make sure package epel-release installed on the guest before running /usr/bin/crb enable as /usr/bin/crb is from package epel-release, right?

happz commented 11 months ago

Hi @happz, thanks very much for clarification! BTW, if user specifies:

prepare:
   how: feature
   crb: enabled

The plugin should make sure package epel-release installed on the guest before running /usr/bin/crb enable as /usr/bin/crb is from package epel-release, right?

Maybe. Can't we enable CRB without EPEL and /usr/bin/crb?

BTW don't take just my word for all of this, please, wait for wiser and more experienced folks like @psss or @lukaszachy to share their opinions before you start cutting code to appease my views ;)

psss commented 11 months ago

So in the tmt install section we currently have this:

sudo dnf config-manager --enable crb         # CentOS 9
sudo dnf config-manager --enable rhel-CRB    # RHEL 9
sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm

This seems to be working just fine and I haven't heard any problems with just enabling the crb repo. @idorax, what made you think that /usr/bin/crb should be used?

happz commented 11 months ago

Hm, one extra note, also for @thrix who wants to operate in the area, working on the all-mighty install plugin: we really, really need an abstraction layer for "package manager" operations. We have prepare/install, we have prepare/feature, we have tmt core installing rsync, we will have checks that will require some binaries to be installed on a guest, in the same way tmt already needs rsync today. All these parts implement their own checks on whether a package exists or listing repos, and staying on this course would be a route to hell. I'll try to coordinate work on liberating the existing implementations from prepare/install, and come up with some shared library with repolist() and install_packages() primitives.

idorax commented 11 months ago

This seems to be working just fine and I haven't heard any problems with just enabling the crb repo. @idorax, what made you think that /usr/bin/crb should be used?

Hi @psss, after command

sudo dnf -y install \
    https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

is run on rhel8, such info in the following is seen from the output,

...<snip>...
  Installing       : epel-release-8-19.el8.noarch                                                                                             1/1 
  Running scriptlet: epel-release-8-19.el8.noarch                                                                                             1/1 
Many EPEL packages require the CodeReady Builder (CRB) repository.
It is recommended that you run /usr/bin/crb enable to enable the CRB repository.
...<snip>...

So I once thought /usr/bin/crb enable was the way to enable the CRB repos.

I'll use the dnf config-manager --enable crb ..., thank you for your clarification!

psss commented 11 months ago

Ah, I see. So this seems to be an alternative way how to enable the crb repo.

I'll use the dnf config-manager --enable crb ..., thank you for your clarification!

Sure!

idorax commented 10 months ago

So in the tmt install section we currently have this:

sudo dnf config-manager --enable crb         # CentOS 9
sudo dnf config-manager --enable rhel-CRB    # RHEL 9
sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm

Hi @psss, for enabling/disabling crb, I looked into it on RHEL 9, but failed to run sudo dnf config-manager --enable rhel-CRB. So, before invoking sudo dnf config-manager --enable rhel-CRB, do we need to install a package which supports to create rhel-CRB repo or something else should be taken care of? 

root# cat /etc/*release | grep 'PRETTY_NAME'
PRETTY_NAME="Red Hat Enterprise Linux 9.0 (Plow)"
root#  dnf config-manager --enable rhel-CRB
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

Error: No matching repo to modify: rhel-CRB.
root#  echo $?
1
root# grep -Ei 'crb' /etc/yum.repos.d/*.repo | awk -F':' '{print $1}' | uniq
/etc/yum.repos.d/beaker-CRB-debuginfo.repo
/etc/yum.repos.d/beaker-CRB.repo
root# rpm -qf /etc/yum.repos.d/beaker-CRB*.repo
file /etc/yum.repos.d/beaker-CRB.repo is not owned by any package
file /etc/yum.repos.d/beaker-CRB-debuginfo.repo is not owned by any package

Looks files /etc/yum.repos.d/beaker-CRB*.repo are created without installing any package. If we look into the baseurl of x86_64, there are a lot of packages for CRB. Right here I'm a bit confused, please help to clarify, thanks very much!

happz commented 10 months ago

...

Looks files /etc/yum.repos.d/beaker-CRB*.repo are created without installing any package. If we look into the baseurl of x86_64, there are a lot of packages for CRB. Right here I'm a bit confused, please help to clarify, thanks very much!

The old curse of Beaker vs. the rest of the world. Testing Farm, and BaseOS CI, take care of presenting repositories with unified names. But, you can easily run into beaker-* repositories when provisioning on your own. I do not have any immediate advice, stuff like this should probably be part of CI system profile and tmt could ask questions like "Hey, how are repositories called in your environment? What's the name of the CRB repo?", but we're not there yet...

Maybe we just need to put more work into this, and instead of a simple one-liner shell script, do some discovery until we make CI systems responsible for this info & relieve tmt from guessing :/ I don't know, would something like this help? Just throwing out some ideas, test a few well-known alternatives, and give up if everything fails...

dnf repolist > repolist.txt

grep beaker-CRB repolist.txt
if [ "$?" = "0"]; then
    dnf config-manager --enable beaker-CRB
    exit 0
fi

grep rhel-CRB repolist.txt
if [ "$?" = "0"]; then
    dnf config-manager --enable rhel-CRB
    exit 0
fi

...
idorax commented 10 months ago

Copy & paste the latest solution,

Q: prepare/feature - plenty of enable repo/install package scripts, URLs, repo names, 
    hardcoded - how can we make it easier to maintain and update when needed?
A: Ansible is the answer
   o ansible-core as a tmt dependency
   o use ansible facts as a source for guest facts (later)
   o implement tmt feature using ansible playbooks
     + Allow user to override which playbook is run
idorax commented 10 months ago

Hi @happz, since we will use solution based on ansible, I had a try to write a playbook for rhel7 to enable repos related to epel, 

# enable-epel-rhel7.yaml
- hosts: guest-rhel7
  user: root
  tasks:
    - name: Install package 'epel-release'
      ansible.builtin.yum:
        name: https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm 
        state: present
    - name: Install package 'yum-utils'
      ansible.builtin.yum:
        name: yum-utils
        state: present
    - name: Enable repos related to 'epel'
      shell: yum-config-manager --enable epel epel-debuginfo epel-source

Then run it via ansible-playbook, which works fine. 

root@FOO:sandbox# cat hosts
[guest-rhel7]
foo.redhat.com

root@FOO:sandbox# ansible-playbook -i hosts enable-epel-rhel7.yaml 
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details

PLAY [guest-rhel7] *******************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************
ok: [foo.redhat.com]

TASK [Install package 'epel-release'] ************************************************************************************************************
ok: [foo.redhat.com]

TASK [Install package 'yum-utils'] ***************************************************************************************************************
ok: [foo.redhat.com]

TASK [Enable repos 'epel'] ***********************************************************************************************************************
changed: [foo.redhat.com]

PLAY RECAP ***************************************************************************************************************************************
foo.redhat.com : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

But I don't know how to embed the playbook script and guest (e.g. its IP or hostname) into our TMT code, any suggestion? Thanks!

happz commented 10 months ago

Hi! I hope you don't feel bad about the change of course - I believe this will be for the best, making the plugin easier to maintain and extend.


# enable-epel-rhel7.yaml

I'd probably reorder the components here: since "epel" is the feature, it should be the first one, then I'd follow with "enable" or "disable", followed by the distro name. epel-enable-rhel7.yaml.

Note that we can use directories if needed, and epel/enable-rhel7.yaml would also be a nice path from the point of someone who expects more than just a few of these files.

The second point would be the rhel7 - here we should aim for something the plugin can extract from guest facts, e.g. combination of ID and VERSION_ID from os_release_content. The idea here would be the plugin would construct the name from the feature, direction and facts, and some kind of mapping through a dictionary ("Fedora Linux 39" => "fedora38") should be the last resort. For now, rhel7 would do, but later we should go with a declarative approach.

We can even support multiple options: does ${feature}-${enable}-${os_release[ID]}${os_release[VERSION_ID]}.yaml exist? It does? Good, run it. It doesn't? How about slightly verbose name then, ${feature}-${enable}-${os_release[NAME]}${os_release[VERSION]}.yaml? It does exist? Run it. It doesn't? Well, we give up, let user know we don't know how to handle this guest.

  • hosts: guest-rhel7

This bit should be much more generic, "rhel-7" is pointless here, try all. It will simply run for the given guest, any given guest.

user: root tasks:

Then run it via ansible-playbook, which works fine.

root@FOO:sandbox# cat hosts
[guest-rhel7]
foo.redhat.com

root@FOO:sandbox# ansible-playbook -i hosts enable-epel-rhel7.yaml 
...

You can it this way, to avoid the need for a file-based inventory: ansible-playbook -ifoo.redhat.com, enable-epel-rhel7.yaml - not the trailing comma after the hostname, it's important and must be present.

But I don't know how to embed the playbook script and guest (e.g. its IP or hostname) into our TMT code, any suggestion? Thanks!

Yep, we will need to dump guest facts and its properties into a YAML file we would then feel Ansible via --extra-vars @filename. There should be a method in Guest class which would save something like {'role': self.role, 'name': self.name, 'facts': self.facts.to_serialized()} into a given file. The file name would be provided by the plugin when calling guest.ansible() with the playbook - the filename must be unique since the plugin might be running for different guests at the same time. See filename_base in prepare/shell, for example.

The file does not have to be pushed to guest though, as long as it exists on the tmt runner and Guest.ansible() can reach it. IIRC, self.ansible(extra_args='@the_constructed_filepath') would be the way to pass it to ansible runner in our code.