Question: is_installed was changed for Debian and an installed package cannot be found after the change

mdelapenya commented 6 years ago

Hi, I'm finding differences running my molecules tests on Travis since the latest version of testinfra. My tests passed in 1.12, but now they are failing in 1.14.

I'm sure the package is installed in the machine, but the python tests says the opposite. Or maybe I'm wrong 🤔 Could you please help me with this issue?

Thanks!

Here I share my scenario:

Ansible task in my role:

- name: "Install jq and all GIT packages"
  become: "yes"
  apt:
    name: "{{ item.name }}"
    state: "{{ item.state }}"
  changed_when: false
  with_items:
    - { name: "jq", state: "latest" }
    - { name: "git-all", state: "latest" }

This is the output of the test execution:

    TASK [jenkins-slave : Install jq and all GIT packages] *************************
    ok: [instance] => (item={u'state': u'latest', u'name': u'jq'})
    ok: [instance] => (item={u'state': u'latest', u'name': u'git-all'})

And the output of the idempotence check:

--> Action: 'idempotence'
Idempotence completed successfully.

This is the python test in molecule:

def test_git_all_is_installed(host):
    assert host.package("git-all").is_installed

The output of the failing test result:

=================================== FAILURES ===================================
    ________________ test_git_all_is_installed[ansible://instance] _________________

    host = <testinfra.host.Host object at 0x7fa2223b61d0>

        def test_git_all_is_installed(host):
    >       assert host.package("git-all").is_installed
    E       AssertionError: assert False
    E        +  where False = <package git-all>.is_installed
    E        +    where <package git-all> = <class 'testinfra.modules.base.DebianPackage'>('git-all')
    E        +      where <class 'testinfra.modules.base.DebianPackage'> = <testinfra.host.Host object at 0x7fa2223b61d0>.package

    tests/test_default.py:29: AssertionError
    ------------------------------ Captured log call -------------------------------
    ansible.py                  61 INFO     RUN Ansible(u'shell', u"dpkg-query -f '${Status}' -W git-all", {}): {'_ansible_no_log': False,
     '_ansible_parsed': True,
     u'changed': True,
     u'cmd': u"dpkg-query -f '${Status}' -W git-all",
     u'delta': u'0:00:00.042013',
     u'end': u'2018-07-10 23:04:28.901809',
     u'invocation': {u'module_args': {u'_raw_params': u"dpkg-query -f '${Status}' -W git-all",
                                      u'_uses_shell': True,
                                      u'chdir': None,
                                      u'creates': None,
                                      u'executable': None,
                                      u'removes': None,
                                      u'stdin': None,
                                      u'warn': True}},
     u'msg': u'non-zero return code',
     u'rc': 1,
     u'start': u'2018-07-10 23:04:28.859796',
     u'stderr': u'dpkg-query: no packages found matching git-all',
     'stderr_lines': [u'dpkg-query: no packages found matching git-all'],
     u'stdout': u'',
     'stdout_lines': []}
    base.py                    241 INFO     RUN CommandResult(command=u"dpkg-query -f '${Status}' -W git-all", exit_status=1, stdout=u'', stderr=u'dpkg-query: no packages found matching git-all')
    ===================== 1 failed, 9 passed in 67.93 seconds ======================

mdelapenya commented 6 years ago

By the way, I have other 4 tests where I check for packages using is_installed, and all of them works. So maybe it could be narrowed down to the git-all package...

mdelapenya commented 6 years ago

By the way, using git instead of git-all fixed the issue, so I believe the problem is located in the git-all package

codylane commented 6 years ago

Hmm, very interesting, can you please provide the version of debian you are using? I just tested this on debian stretch and I do not get this error. I also wonder what version of ansible you are using, but I don't think this is a testinfra bug.

This is not the correct forum to discuss ansible or molecule problems but I suspect your problem might be in your playbook/role and not with testinfra itself. I could be wrong too but I currently cannot reproduce this error using the latest version of ansible or testinfra and I didn't even try to use molecule.

I suspect your problem might be that you have changed_when: false which I've never seen anyone do before on a package install and I can only assume this was a copy pasta mistake but again without knowing more details of your environment it's hard to say why you got this error.

Creating a symlink for the inventory

ln -s .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory inventory

`cat test_packages.py`

#!/usr/bin/env python

# import pytest
# import testinfra
import os

BASE_DIR = os.path.abspath(os.path.dirname(__file__))
INVENTORY = os.path.join(BASE_DIR, 'inventory')
testinfra_hosts = ('ansible://default?ansible_inventory=' + INVENTORY,)

def test_git_all_is_installed(host):
    assert host.package("git-all").is_installed

cat Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  config.vm.define 'default' do |config|
    config.vm.box = "debian/stretch64"
    config.vm.hostname = "localhost"

    # Vagrant >= 1.7 wants to replace the insecure_key with public boxes, but
    # there is a bug in that implentation so we just allow the insecure_key
    # anyway.
    config.ssh.insert_key = false

    config.vm.box_check_update = false

    config.vm.network "private_network", ip: "67.77.255.22"

    config.vm.synced_folder ".", "/vagrant",
      disabled: false,
      type: "sshfs",
      ssh_opts_append: "-o Compression=yes -o ControlPersist=60s -o ControlMaster=auto",
      sshfs_opts_append: "-o cache=no -o nonempty"

    # config.vm.provision "shell", path: "install.sh"

    config.vm.provision "ansible" do |ansible|
      ansible.verbose = "v"
      ansible.playbook = "playbook.yml"
    end

    config.vm.provider "virtualbox" do |vb|
      #  # Display the VirtualBox GUI when booting the machine
      #  vb.gui = true

      vb.cpus = "1"
      vb.memory = "768"

      vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
      vb.customize ["modifyvm", :id, "--natdnsproxy1", "on"]
    end
  end
end

cat playbook.yml

---

- hosts: default
  become: yes

  tasks:
    - name: "Install jq and all GIT packages"
      become: "yes"
      apt:
        name: "{{ item.name }}"
        state: "{{ item.state }}"
      changed_when: false
      with_items:
        - { name: "jq", state: "latest" }
        - { name: "git-all", state: "latest" }
      when:
        ansible_os_family == 'Debian'

cat ansible.cfg

[ssh_connection]
pipelining = True

[defaults]
inventory= ./inventory
error_on_missing_handler = True
ansible_managed = Ansible managed: {file} modified on %Y-%m-%d %H:%M:%S by {uid} on {host}
deprecation_warnings = True
display_skipped_hosts = True
host_key_checking = False

gathering = smart
# gather_subset = all
fact_caching = jsonfile
fact_caching_connection = tmp/ansible-facts
fact_caching_timeout = 60

cat test-requirements.txt

ansible==2.6.1
asn1crypto==0.24.0
atomicwrites==1.1.5
attrs==18.1.0
bcrypt==3.1.4
cffi==1.11.5
cryptography==2.2.2
enum34==1.1.6
funcsigs==1.0.2
idna==2.7
ipaddress==1.0.22
Jinja2==2.10
MarkupSafe==1.0
more-itertools==4.2.0
paramiko==2.4.1
pluggy==0.6.0
py==1.5.4
pyasn1==0.4.3
pycparser==2.18
PyNaCl==1.2.1
pytest==3.6.3
PyYAML==3.13
six==1.11.0
testinfra==1.14.0

Create a new virtual env

virtualenv -p python2.7 venv
. venv/bin/activate
pip install -r test-requirements.txt

Run vagrant and provision the vm

vagrant up

# run the tests
$ pytest -vvs
=============================================================================================== test session starts ===============================================================================================
platform darwin -- Python 2.7.15, pytest-3.6.3, py-1.5.4, pluggy-0.6.0 -- /private/tmp/xenial-test/venv/bin/python2.7
cachedir: .pytest_cache
rootdir: /private/tmp/xenial-test, inifile:
plugins: testinfra-1.14.0
collected 1 item                                                                                                                                                                                                  

test_packages.py::test_git_all_is_installed[ansible://default] INFO:testinfra:RUN Ansible(u'shell', u'uname -s', {}): {'_ansible_no_log': False,
 '_ansible_parsed': True,
 u'changed': True,
 u'cmd': u'uname -s',
 u'delta': u'0:00:00.002308',
 u'end': u'2018-07-17 05:31:12.045883',
 u'invocation': {u'module_args': {u'_raw_params': u'uname -s',
                                  u'_uses_shell': True,
                                  u'argv': None,
                                  u'chdir': None,
                                  u'creates': None,
                                  u'executable': None,
                                  u'removes': None,
                                  u'stdin': None,
                                  u'warn': True}},
 u'rc': 0,
 u'start': u'2018-07-17 05:31:12.043575',
 u'stderr': u'',
 'stderr_lines': [],
 u'stdout': u'Linux',
 'stdout_lines': [u'Linux']}
INFO:testinfra:RUN CommandResult(command=u'uname -s', exit_status=0, stdout=u'Linux', stderr=u'')
INFO:testinfra:RUN Ansible(u'shell', u'lsb_release -a', {}): {'_ansible_no_log': False,
 '_ansible_parsed': True,
 u'changed': True,
 u'cmd': u'lsb_release -a',
 u'delta': u'0:00:00.038283',
 u'end': u'2018-07-17 05:31:12.282728',
 u'invocation': {u'module_args': {u'_raw_params': u'lsb_release -a',
                                  u'_uses_shell': True,
                                  u'argv': None,
                                  u'chdir': None,
                                  u'creates': None,
                                  u'executable': None,
                                  u'removes': None,
                                  u'stdin': None,
                                  u'warn': True}},
 u'rc': 0,
 u'start': u'2018-07-17 05:31:12.244445',
 u'stderr': u'No LSB modules are available.',
 'stderr_lines': [u'No LSB modules are available.'],
 u'stdout': u'Distributor ID:\tDebian\nDescription:\tDebian GNU/Linux 9.4 (stretch)\nRelease:\t9.4\nCodename:\tstretch',
 'stdout_lines': [u'Distributor ID:\tDebian',
                  u'Description:\tDebian GNU/Linux 9.4 (stretch)',
                  u'Release:\t9.4',
                  u'Codename:\tstretch']}
INFO:testinfra:RUN CommandResult(command=u'lsb_release -a', exit_status=0, stdout=u'Distributor ID:\tDebian\nDescription:\tDebian GNU/Linux 9.4 (stretch)\nRelease:\t9.4\nCodename:\tstretch', stderr=u'No LSB modules are available.')
INFO:testinfra:RUN Ansible(u'shell', u'command -v dpkg-query', {}): {'_ansible_no_log': False,
 '_ansible_parsed': True,
 u'changed': True,
 u'cmd': u'command -v dpkg-query',
 u'delta': u'0:00:00.001788',
 u'end': u'2018-07-17 05:31:12.467429',
 u'invocation': {u'module_args': {u'_raw_params': u'command -v dpkg-query',
                                  u'_uses_shell': True,
                                  u'argv': None,
                                  u'chdir': None,
                                  u'creates': None,
                                  u'executable': None,
                                  u'removes': None,
                                  u'stdin': None,
                                  u'warn': True}},
 u'rc': 0,
 u'start': u'2018-07-17 05:31:12.465641',
 u'stderr': u'',
 'stderr_lines': [],
 u'stdout': u'/usr/bin/dpkg-query',
 'stdout_lines': [u'/usr/bin/dpkg-query']}
INFO:testinfra:RUN CommandResult(command=u'command -v dpkg-query', exit_status=0, stdout=u'/usr/bin/dpkg-query', stderr=u'')
INFO:testinfra:RUN Ansible(u'shell', u"dpkg-query -f '${Status}' -W git-all", {}): {'_ansible_no_log': False,
 '_ansible_parsed': True,
 u'changed': True,
 u'cmd': u"dpkg-query -f '${Status}' -W git-all",
 u'delta': u'0:00:00.010250',
 u'end': u'2018-07-17 05:31:12.658868',
 u'invocation': {u'module_args': {u'_raw_params': u"dpkg-query -f '${Status}' -W git-all",
                                  u'_uses_shell': True,
                                  u'argv': None,
                                  u'chdir': None,
                                  u'creates': None,
                                  u'executable': None,
                                  u'removes': None,
                                  u'stdin': None,
                                  u'warn': True}},
 u'rc': 0,
 u'start': u'2018-07-17 05:31:12.648618',
 u'stderr': u'',
 'stderr_lines': [],
 u'stdout': u'install ok installed',
 'stdout_lines': [u'install ok installed']}
INFO:testinfra:RUN CommandResult(command=u"dpkg-query -f '${Status}' -W git-all", exit_status=0, stdout=u'install ok installed', stderr=u'')
PASSED

mdelapenya commented 6 years ago

Hey, thanks for such a detailed bugtrack 😊

My base OS is Ubuntu 16.04, this is something I did not add to the scenario. I set that at molecule.yml level:

---
dependency:
  name: galaxy
driver:
  name: docker
lint:
  name: yamllint
platforms:
  - name: instance
    image: ubuntu:16.04
provisioner:
  name: ansible
  lint:
    name: ansible-lint
scenario:
  name: default
verifier:
  name: testinfra
  lint:
    name: flake8

codylane commented 6 years ago

I tested this with molecule using your sample configuration and I still get passing tests. I'm fairly confident that the error you are getting is problems with your role + molecule configuration versus a bug with testinfra.

Perhaps you can create a gist of your example to show us how you have your environment setup so that we can try and reproduce the error using your gist?

I've personally tested this both with Vagrant, Docker and molecule and I cannot reproduce your error via testinfra. I get the error when the package is not installed which is the correct behavior of testinfra.

I'm just a contributor to testinfra and can say that I believe testinfra is behaving correctly in this case and wasn't in 1.12.0. Just my $2.

pytest-dev / pytest-testinfra