Activation key tests fail

furhouse commented 5 years ago

SUMMARY

The test/test_playbooks/activation_key.yml playbook seems to fail.

ISSUE TYPE

Bug Report

ANSIBLE VERSION

❯ ansible --version
ansible 2.7.6
  config file = /home/furhouse/foreman-ansible-modules/ansible.cfg
  configured module search path = ['/home/furhouse/foreman-ansible-modules/modules']
  ansible python module location = /usr/lib/python3.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.7.2 (default, Jan 10 2019, 23:51:51) [GCC 8.2.1 20181127]

KATELLO/FOREMAN VERSION

[root@fk1 ~]# rpm -q tfm-rubygem-katello foreman
tfm-rubygem-katello-3.10.0-1.el7.noarch
foreman-1.20.1-1.el7.noarch

NAILGUN VERSION

❯ pip show nailgun | grep Version
Version: 0.31.0

STEPS TO REPRODUCE

~
❯ cd /tmp

/tmp
❯ git clone https://github.com/theforeman/foreman-ansible-modules.git
Cloning into 'foreman-ansible-modules'...
remote: Enumerating objects: 189, done.
remote: Counting objects: 100% (189/189), done.
remote: Compressing objects: 100% (81/81), done.
remote: Total 2357 (delta 132), reused 149 (delta 108), pack-reused 2168
Receiving objects: 100% (2357/2357), 2.15 MiB | 3.82 MiB/s, done.
Resolving deltas: 100% (1814/1814), done.

/tmp
❯ cd foreman-ansible-modules/test/test_playbooks

/tmp/foreman-ansible-modules/test/test_playbooks master
❯ cp /tmp/server_vars.yml .

/tmp/foreman-ansible-modules/test/test_playbooks master
❯ ../..

/tmp/foreman-ansible-modules master
❯ make test_activation_key
pytest 'test/test_crud.py::test_crud[activation_key]'
=============================================================================== test session starts ===============================================================================
platform linux -- Python 3.7.2, pytest-4.0.0, py-1.7.0, pluggy-0.8.0
rootdir: /tmp/foreman-ansible-modules, inifile:
collected 1 item                                                                                                                                                                  

test/test_crud.py F                                                                                                                                                         [100%]

==================================================================================== FAILURES =====================================================================================
____________________________________________________________________________ test_crud[activation_key] ____________________________________________________________________________

module = 'activation_key', record = False

    @pytest.mark.parametrize('module', MODULES)
    def test_crud(module, record):
>       assert run_playbook_vcr(module, record=record) == 0
E       AssertionError: assert 2 == 0
E        +  where 2 = run_playbook_vcr('activation_key', record=False)

test/test_crud.py:80: AssertionError
------------------------------------------------------------------------------ Captured stdout call -------------------------------------------------------------------------------

PLAY [fixtures] ****************************************************************
skipping: no hosts matched

PLAY [tests] *******************************************************************

TASK [Load server config] ******************************************************
ok: [tests]

TASK [Create/update katello activation key] ************************************
changed: [tests]

TASK [fail] ********************************************************************
skipping: [tests]

TASK [Create/update minimal katello activation key] ****************************
ok: [tests]

TASK [fail] ********************************************************************
skipping: [tests]

TASK [Create/update katello activation key] ************************************
changed: [tests]

TASK [fail] ********************************************************************
fatal: [tests]: FAILED! => {"changed": false, "msg": "Creating/updating activation key failed! (expected_change: False)"}

PLAY RECAP *********************************************************************
tests                      : ok=4    changed=2    unreachable=0    failed=1   

================================================================================ warnings summary =================================================================================
/usr/lib/python3.7/site-packages/jinja2/utils.py:485
  /usr/lib/python3.7/site-packages/jinja2/utils.py:485: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import MutableMapping

/usr/lib/python3.7/site-packages/jinja2/runtime.py:318
  /usr/lib/python3.7/site-packages/jinja2/runtime.py:318: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping

/usr/lib/python3.7/site-packages/ansible/errors/__init__.py:22
  /usr/lib/python3.7/site-packages/ansible/errors/__init__.py:22: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Sequence

/usr/lib/python3.7/site-packages/yaml/constructor.py:126
  /usr/lib/python3.7/site-packages/yaml/constructor.py:126: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    if not isinstance(key, collections.Hashable):

/usr/lib/python3.7/site-packages/ansible/utils/unsafe_proxy.py:56
  /usr/lib/python3.7/site-packages/ansible/utils/unsafe_proxy.py:56: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping, MutableSequence, Set
  /usr/lib/python3.7/site-packages/ansible/utils/unsafe_proxy.py:56: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping, MutableSequence, Set

/usr/lib/python3.7/site-packages/ansible/utils/plugin_docs.py:23
  /usr/lib/python3.7/site-packages/ansible/utils/plugin_docs.py:23: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import MutableMapping, MutableSet, MutableSequence

/usr/lib/python3.7/site-packages/ansible/utils/listify.py:22
  /usr/lib/python3.7/site-packages/ansible/utils/listify.py:22: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Iterable

-- Docs: https://docs.pytest.org/en/latest/warnings.html
====================================================================== 1 failed, 8 warnings in 1.70 seconds =======================================================================
make: *** [Makefile:32: test_activation_key] Error 1

/tmp/foreman-ansible-modules master
❯ make record_activation_key
rm -f test/test_playbooks/fixtures/activation_key-*.yml
pytest 'test/test_crud.py::test_crud[activation_key]' --record
=============================================================================== test session starts ===============================================================================
platform linux -- Python 3.7.2, pytest-4.0.0, py-1.7.0, pluggy-0.8.0
rootdir: /tmp/foreman-ansible-modules, inifile:
collected 1 item                                                                                                                                                                  

test/test_crud.py F                                                                                                                                                         [100%]

==================================================================================== FAILURES =====================================================================================
____________________________________________________________________________ test_crud[activation_key] ____________________________________________________________________________

module = 'activation_key', record = True

    @pytest.mark.parametrize('module', MODULES)
    def test_crud(module, record):
>       assert run_playbook_vcr(module, record=record) == 0
E       AssertionError: assert 2 == 0
E        +  where 2 = run_playbook_vcr('activation_key', record=True)

test/test_crud.py:80: AssertionError
------------------------------------------------------------------------------ Captured stdout call -------------------------------------------------------------------------------

PLAY [fixtures] ****************************************************************

TASK [Load server config] ******************************************************
ok: [fixtures]

TASK [Ensure organization 'Test Organization' is present] **********************
changed: [fixtures]

TASK [fail] ********************************************************************
skipping: [fixtures]

TASK [Ensure product 'Test Product' is present] ********************************
changed: [fixtures]

TASK [fail] ********************************************************************
skipping: [fixtures]

TASK [Create katello repository] ***********************************************
changed: [fixtures]

PLAY [tests] *******************************************************************

TASK [Load server config] ******************************************************
ok: [tests]

TASK [Create/update katello activation key] ************************************
changed: [tests]

TASK [fail] ********************************************************************
skipping: [tests]

TASK [Create/update minimal katello activation key] ****************************
ok: [tests]

TASK [fail] ********************************************************************
skipping: [tests]

TASK [Create/update katello activation key] ************************************
changed: [tests]

TASK [fail] ********************************************************************
fatal: [tests]: FAILED! => {"changed": false, "msg": "Creating/updating activation key failed! (expected_change: False)"}

PLAY RECAP *********************************************************************
fixtures                   : ok=4    changed=3    unreachable=0    failed=0   
tests                      : ok=4    changed=2    unreachable=0    failed=1   

================================================================================ warnings summary =================================================================================
/usr/lib/python3.7/site-packages/jinja2/utils.py:485
  /usr/lib/python3.7/site-packages/jinja2/utils.py:485: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import MutableMapping

/usr/lib/python3.7/site-packages/jinja2/runtime.py:318
  /usr/lib/python3.7/site-packages/jinja2/runtime.py:318: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping

/usr/lib/python3.7/site-packages/ansible/errors/__init__.py:22
  /usr/lib/python3.7/site-packages/ansible/errors/__init__.py:22: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Sequence

/usr/lib/python3.7/site-packages/yaml/constructor.py:126
  /usr/lib/python3.7/site-packages/yaml/constructor.py:126: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    if not isinstance(key, collections.Hashable):

/usr/lib/python3.7/site-packages/ansible/utils/unsafe_proxy.py:56
  /usr/lib/python3.7/site-packages/ansible/utils/unsafe_proxy.py:56: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping, MutableSequence, Set
  /usr/lib/python3.7/site-packages/ansible/utils/unsafe_proxy.py:56: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping, MutableSequence, Set

/usr/lib/python3.7/site-packages/ansible/utils/plugin_docs.py:23
  /usr/lib/python3.7/site-packages/ansible/utils/plugin_docs.py:23: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import MutableMapping, MutableSet, MutableSequence

/usr/lib/python3.7/site-packages/ansible/utils/listify.py:22
  /usr/lib/python3.7/site-packages/ansible/utils/listify.py:22: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Iterable

-- Docs: https://docs.pytest.org/en/latest/warnings.html
====================================================================== 1 failed, 8 warnings in 12.27 seconds ======================================================================
make: *** [Makefile:36: record_activation_key] Error 1

EXPECTED RESULTS

I would expect the test_activation_key target to pass.

ACTUAL RESULTS

Both the test_activation_key and record_activation_key targets fail.

NOTES

Just quickly disabling some tasks/vars with expected_change: false seems to 'help':

diff --git a/test/test_playbooks/activation_key.yml b/test/test_playbooks/activation_key.yml
index 6e85d03..2d4f114 100644
--- a/test/test_playbooks/activation_key.yml
+++ b/test/test_playbooks/activation_key.yml
@@ -27,28 +27,28 @@
   - include: tasks/activation_key_minimal.yml
     vars:
       expected_change: false
-  - include: tasks/activation_key.yml
-    vars:
-      expected_change: false
-  - include: tasks/activation_key.yml
-    vars:
-      activation_key_auto_attach: True
-      expected_change: true
+  # - include: tasks/activation_key.yml
+  #   vars:
+  #     expected_change: false
   - include: tasks/activation_key.yml
     vars:
       activation_key_auto_attach: True
-      expected_change: false
-  - include: tasks/activation_key.yml
-    vars:
-      activation_key_auto_attach: False
-      activation_key_state: present_with_defaults
-      expected_change: false
+  #     expected_change: true
+  # - include: tasks/activation_key.yml
+  #   vars:
+  #     activation_key_auto_attach: True
+  #     expected_change: false
+  # - include: tasks/activation_key.yml
+  #   vars:
+  #     activation_key_auto_attach: False
+  #     activation_key_state: present_with_defaults
+  #     expected_change: false
   - include: tasks/activation_key_copy.yml
     vars:
-      expected_change: true
-  - include: tasks/activation_key_copy.yml
-    vars:
-      expected_change: false
+  #     expected_change: true
+  # - include: tasks/activation_key_copy.yml
+  #   vars:
+  #     expected_change: false
   - include: tasks/activation_key.yml
     vars:
       activation_key_state: absent

❯ make test_activation_key
pytest 'test/test_crud.py::test_crud[activation_key]'
========================================================================================================================================================================================================== test session starts ===========================================================================================================================================================================================================
platform linux -- Python 3.7.2, pytest-4.0.0, py-1.7.0, pluggy-0.8.0
rootdir: /tmp/foreman-ansible-modules, inifile:
collected 1 item                                                                                                                                                                                                                                                                                                                                                                                                                         

test/test_crud.py .                                                                                                                                                                                                                                                                                                                                                                                                                [100%]

============================================================================================================================================================================================================ warnings summary ============================================================================================================================================================================================================
/usr/lib/python3.7/site-packages/jinja2/utils.py:485
  /usr/lib/python3.7/site-packages/jinja2/utils.py:485: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import MutableMapping

/usr/lib/python3.7/site-packages/jinja2/runtime.py:318
  /usr/lib/python3.7/site-packages/jinja2/runtime.py:318: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping

/usr/lib/python3.7/site-packages/ansible/errors/__init__.py:22
  /usr/lib/python3.7/site-packages/ansible/errors/__init__.py:22: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Sequence

/usr/lib/python3.7/site-packages/yaml/constructor.py:126
  /usr/lib/python3.7/site-packages/yaml/constructor.py:126: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    if not isinstance(key, collections.Hashable):

/usr/lib/python3.7/site-packages/ansible/utils/unsafe_proxy.py:56
  /usr/lib/python3.7/site-packages/ansible/utils/unsafe_proxy.py:56: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping, MutableSequence, Set
  /usr/lib/python3.7/site-packages/ansible/utils/unsafe_proxy.py:56: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping, MutableSequence, Set

/usr/lib/python3.7/site-packages/ansible/utils/plugin_docs.py:23
  /usr/lib/python3.7/site-packages/ansible/utils/plugin_docs.py:23: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import MutableMapping, MutableSet, MutableSequence

/usr/lib/python3.7/site-packages/ansible/utils/listify.py:22
  /usr/lib/python3.7/site-packages/ansible/utils/listify.py:22: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Iterable

-- Docs: https://docs.pytest.org/en/latest/warnings.html
================================================================================================================================================================================================== 1 passed, 8 warnings in 2.68 seconds ==================================================================================================================================================================================================

sean797 commented 5 years ago

Thanks @furhouse I think this may be your problem:


pip show nailgun | grep Version
Version: 0.31.0

I'd suggest trying with Nailgun on master or the 6.4.z branch (as I've backported a few commits on that branch)

furhouse commented 5 years ago

To make sure I'm using python3, I've set up a venv, then explicitly installed nailgun from both master and 6.4.z. make test_activation_keypasses with both branches, record_activation_key still seems to fail:

❯ virtualenv -p /usr/bin/python3.6 --system-site-packages venv2

❯ source venv2/bin/activate

venv2 ❯ pip install 'git+https://github.com/SatelliteQE/nailgun.git@6.4.z#egg=nailgun' / also tested master
...
Successfully installed nailgun-0.30.2

venv2 ❯ cd /tmp/foreman-ansible-modules

venv2 ❯ make test-setup

venv2 ❯ ansible --version
ansible 2.7.7
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/furhouse/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /tmp/venv2/lib/python3.6/site-packages/ansible
  executable location = /tmp/venv2/bin/ansible
  python version = 3.6.8 (default, Feb 20 2019, 08:51:39) [GCC 8.2.1 20181127]

venv2 ❯ make test_activation_key
pytest 'test/test_crud.py::test_crud[activation_key]'
============================================================================================================================================================================ test session starts ============================================================================================================================================================================
platform linux -- Python 3.6.8, pytest-4.3.0, py-1.7.0, pluggy-0.8.1
rootdir: /tmp/foreman-ansible-modules, inifile:
collected 1 item                                                                                                                                                                                                                                                                                                                                                            

test/test_crud.py .                                                                                                                                                                                                                                                                                                                                                   [100%]

========================================================================================================================================================================= 1 passed in 7.23 seconds ==========================================================================================================================================================================

venv2 ❯ make record_activation_key
rm -f test/test_playbooks/fixtures/activation_key-*.yml
pytest 'test/test_crud.py::test_crud[activation_key]' --record
============================================================================================================================================================================ test session starts ============================================================================================================================================================================
platform linux -- Python 3.6.8, pytest-4.3.0, py-1.7.0, pluggy-0.8.1
rootdir: /tmp/foreman-ansible-modules, inifile:
collected 1 item                                                                                                                                                                                                                                                                                                                                                            

test/test_crud.py F                                                                                                                                                                                                                                                                                                                                                   [100%]

================================================================================================================================================================================= FAILURES ==================================================================================================================================================================================
_________________________________________________________________________________________________________________________________________________________________________ test_crud[activation_key] _________________________________________________________________________________________________________________________________________________________________________

module = 'activation_key', record = True

    @pytest.mark.parametrize('module', MODULES)
    def test_crud(module, record):
>       assert run_playbook_vcr(module, record=record) == 0
E       AssertionError: assert 2 == 0
E        +  where 2 = run_playbook_vcr('activation_key', record=True)

test/test_crud.py:80: AssertionError
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

PLAY [fixtures] ****************************************************************

TASK [Load server config] ******************************************************
ok: [fixtures]

TASK [Ensure organization 'Test Organization' is present] **********************
ok: [fixtures]

TASK [fail] ********************************************************************
skipping: [fixtures]

TASK [Ensure product 'Test Product' is present] ********************************
ok: [fixtures]

TASK [fail] ********************************************************************
skipping: [fixtures]

TASK [Create katello repository] ***********************************************
ok: [fixtures]

PLAY [tests] *******************************************************************

TASK [Load server config] ******************************************************
ok: [tests]

TASK [Create/update katello activation key] ************************************
changed: [tests]

TASK [fail] ********************************************************************
skipping: [tests]

TASK [Create/update minimal katello activation key] ****************************
ok: [tests]

TASK [fail] ********************************************************************
skipping: [tests]

TASK [Create/update katello activation key] ************************************
changed: [tests]

TASK [fail] ********************************************************************
fatal: [tests]: FAILED! => {"changed": false, "msg": "Creating/updating activation key failed! (expected_change: False)"}

PLAY RECAP *********************************************************************
fixtures                   : ok=4    changed=0    unreachable=0    failed=0   
tests                      : ok=4    changed=2    unreachable=0    failed=1   

========================================================================================================================================================================= 1 failed in 7.99 seconds ==========================================================================================================================================================================
make: *** [Makefile:36: record_activation_key] Error 1

theforeman / foreman-ansible-modules