search

avocado-framework / avocado

Avocado is a set of tools and libraries to help with automated testing. One can call it a test framework with benefits. Native tests are written in Python and they follow the unittest pattern, but any executable can serve as a test.
https://avocado-framework.github.io/
Other
346 stars 343 forks source link

Ansible plugin should require `ansible` instead of `ansible-core` because of debug callback plugin #5638

Closed lkotek closed 1 year ago

lkotek commented 1 year ago

Describe the bug Ansible plugin requires only ansible-core to be installed if avocado is installed in development mode. This is not sufficient in some situations. Callback plugin debug is not available as a part of ansible-core, which may cause some test cases to fail. Setting with ansible_callback_plugin = debug is the default at least for ansible related test cases from tp-qemu repository.

Steps to reproduce Install avocado framework in development mode:

python3 setup.py develop --user

Check if debug callback plugin is available:

ansible-doc -t callback -l | grep debug

Optional: Install other dependencies and run ansible_test case relying on the debug callback plugin.

Expected behavior

  1. Full ansible version is installed no matter devel or install is used for avocado environment preparation.
  2. As a result, debug callback plugin ansible.posix.debug is installed.

Current behavior

  1. Only ansible-core is installed in development mode.
  2. Test cases relying on debug callback plugin fail.

System information (please complete the following information):

Additional information

The behaviour explained above causes problem only in case avocado is installed in development mode before the main ansible test case is executed (test case itself handles ansible installation via ansible provider by checking ansible-playbook command first, but this is provided also via ansible-core).

Also the situation may be eventually workarounded via python3 setup.py develop --skip-optional-plugins --user, because ansible plugin is not strictly needed for the development of ansible related test cases from tp-qemu repository.

We can also add additional logic directly into the ansible provider, but the key point is: can we consider just to simply modify /setup.py#L37 and switch to ansible by default?

We hit this problem also in the past in different situation and workarounded it, but it keeps to occur from time to time.

lkotek commented 1 year ago

Hi @yanan-fu, I am adding you for awareness.

clebergnu commented 1 year ago

Hi @lkotek,

Thanks for reporting this issue. You seem to suggest that we have different behaviors depending on how Avocado is set up, that is:

Just to make it clear, that is not really the case. Currently, all set up modes will depend on ansible-core only. For instance, installation via pip:

# python3 -m pip install avocado-framework-plugin-ansible
Collecting avocado-framework-plugin-ansible
  Downloading avocado_framework_plugin_ansible-101.0-py3-none-any.whl (2.7 kB)
Requirement already satisfied: avocado-framework==101.0 in /usr/local/lib/python3.11/site-packages (from avocado-framework-plugin-ansible) (101.0)
Collecting ansible-core
  Downloading ansible_core-2.14.4-py3-none-any.whl (2.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 12.5 MB/s eta 0:00:00
Requirement already satisfied: setuptools in /usr/local/lib/python3.11/site-packages (from avocado-framework==101.0->avocado-framework-plugin-ansible) (62.6.0)
Collecting jinja2>=3.0.0
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 kB 42.6 MB/s eta 0:00:00
Collecting PyYAML>=5.1
  Downloading PyYAML-6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (757 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 757.9/757.9 kB 79.9 MB/s eta 0:00:00
Collecting cryptography
  Downloading cryptography-40.0.2-cp36-abi3-manylinux_2_28_x86_64.whl (3.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.7/3.7 MB 27.4 MB/s eta 0:00:00
Collecting packaging
  Downloading packaging-23.1-py3-none-any.whl (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 45.7 MB/s eta 0:00:00
Collecting resolvelib<0.9.0,>=0.5.3
  Downloading resolvelib-0.8.1-py2.py3-none-any.whl (16 kB)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.1.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (27 kB)
Collecting cffi>=1.12
  Downloading cffi-1.15.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (462 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 462.6/462.6 kB 29.6 MB/s eta 0:00:00
Collecting pycparser
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 73.0 MB/s eta 0:00:00
Installing collected packages: resolvelib, PyYAML, pycparser, packaging, MarkupSafe, jinja2, cffi, cryptography, ansible-core, avocado-framework-plugin-ansible
Successfully installed MarkupSafe-2.1.2 PyYAML-6.0 ansible-core-2.14.4 avocado-framework-plugin-ansible-101.0 cffi-1.15.1 cryptography-40.0.2 jinja2-3.1.2 packaging-23.1 pycparser-2.21 resolvelib-0.8.1

The same is true for RPM based installations.

Now, while I understand the tp-qemu ansible test need the debug module (and thus extra packages), I fail to see why that has to be a dependency for every user of the Avocado ansible plugin. The ansible-core package, provides IMO, a good number of modules:

add_host.py        debconf.py          git.py              iptables.py       rpm_key.py        systemd_service.py
apt_key.py         debug.py            group_by.py         known_hosts.py    script.py         sysvinit.py
apt.py             dnf.py              group.py            lineinfile.py     service_facts.py  tempfile.py
apt_repository.py  dpkg_selections.py  hostname.py         meta.py           service.py        template.py
assemble.py        expect.py           import_playbook.py  package_facts.py  set_fact.py       unarchive.py
assert.py          fail.py             import_role.py      package.py        set_stats.py      uri.py
async_status.py    fetch.py            import_tasks.py     pause.py          setup.py          user.py
async_wrapper.py   file.py             _include.py         ping.py           shell.py          validate_argument_spec.py
blockinfile.py     find.py             include_role.py     pip.py            slurp.py          wait_for_connection.py
command.py         gather_facts.py     include_tasks.py    raw.py            stat.py           wait_for.py
copy.py            getent.py           include_vars.py     reboot.py         subversion.py     yum.py
cron.py            get_url.py          __init__.py         replace.py        systemd.py        yum_repository.p

While still keeping the dependencies somewhat sane. In my opinion, the dependency on the ansible package should be declared closer to the test or at the very least at the tp-qemu level. Have you considered those options?

Please let me know if I'm missing something, and I'll gladly reconsider my opinion here!

Thanks!!!

lkotek commented 1 year ago

Hi @clebergnu,

Thanks for your reply. I was thinking a lot about my own reply, because actually I understand your point very well. Let me start with one thing I see I wasn't quite clear about:

You seem to suggest that we have different behaviors depending on how Avocado is set up, that is:

* If development mode, `ansible-core` is installed

* If a regular "install" is performed, than the complete ansible package is installed

Just to make it clear, that is not really the case. Currently, all set up modes will depend on ansible-core only.

Sorry I described the problems to be attached only to development mode (but that's just how we hit the issue). Yes, I understand that complete ansible package is never installed.

In our use case, we actually don't need and don't install avocado-framework-plugin-ansible at all (ansible_test doesn't require it), so development mode is the only situation any ansible version is installed together with avocado.

While still keeping the dependencies somewhat sane. In my opinion, the dependency on the ansible package should be declared closer to the test or at the very least at the tp-qemu level. Have you considered those options?

Actually this is exactly what we do via ansible.py#L115. We handle installation of proper ansible at the moment ansible_test case is executed. And we also extended this method to be more flexible recently.

What currently happens is following:

  1. Avocado is installed in development mode (so also ansible-core is installed), then
  2. ansible_test is executed and ansible provider checks for ansible-playbook (which is satisfied via ansible-core),
  3. ansible_test fails due to missing debug plugin.

There are basically two options how to resolve the problem. We can extend our checks directly in ansible provider again (there is already a change introducing it https://github.com/autotest/tp-qemu/pull/3688) or change the dependency of ansible plugin, which is the place where the problem (from our perspective) originates.

But as I said before, I understand your point, especially about keeping the dependencies sane.

I am not sure, if this can change your mind, I just wanted to add the pieces I wasn't clear about before.

richtja commented 1 year ago

Hi @lkotek if you don't use any of optional plugins and avocado-framework-plugin-ansible is making troubles to you. You should be able to disable them by python3 setup.py develop --user --skip-optional-plugins

clebergnu commented 1 year ago

@lkotek thanks for the response, and please let us know if the workaround suggested by @richtja works for you.

lkotek commented 1 year ago

Hi @richtja, thanks for pointing that out. I am aware of it (https://github.com/avocado-framework/avocado/issues/5638#issue-1662380503) :wink: Yes, it is can also resolve the problem.

richtja commented 1 year ago

Hi @richtja, thanks for pointing that out. I am aware of it (#5638 (comment)) wink Yes, it is can also resolve the problem.

Sorry I missed that. But I still think that this will be the easiest solution for this.

I agree with @clebergnu here, that this should be resolved closer to the test. Since this is a problem of develop mode and test environment. IMO, the develop installation is a special type of installation for developers and each of them have different requirements and different environments and we are not able to ensure support of all use-cases. Therefore, if you use develop installation, you have to resolve issues with your environment by yourself and IMO this is exactly an example of such issue.

richtja commented 1 year ago

@lkotek I am closing this issue as resolved. Please feel free to reopen it if you will have any other comets it.

lkotek commented 1 year ago

@richtja Thank you and no worries at all :wink: I confirm we will use --skip-optional-plugins. There was also one more change necessary in our internal automation repository responsible for avocado deployment, but it was minor change.

Thanks you both for help and interesting discussion here! @clebergnu