Tests pass but claims goss failed

[asciifaceman]

Running latest version. ansible 2.7.0 python version = 2.7.10

usage:

- name: Run goss tests
  hosts: all
  roles:
    - { role: degoss, goss_file: tests/goss.yml}

Tests:

user:
  someuser:
    exists: true
    groups:
      - sudo
port:
  tcp:22:
    listening: true
service:
  sshd:
    enabled: true
    running: true
process:
  sshd:
    running: true

PLAY [Run goss tests] **********************************************************

TASK [Gathering Facts] *********************************************************
ok: [default]

TASK [degoss : create workdir] *************************************************
ok: [default]

TASK [degoss : establish facts] ************************************************
ok: [default]

TASK [degoss : include_tasks] **************************************************
included: /Users/ccorbett/redactaroni/ansible-roles/common/roles/degoss/tasks/versions/latest.yml for default

TASK [degoss : detect latest version] ******************************************
ok: [default]

TASK [degoss : establish regular expressions] **********************************
ok: [default]

TASK [degoss : set detected version] *******************************************
ok: [default]

TASK [degoss : include_tasks] **************************************************
skipping: [default]

TASK [degoss : create goss directories] ****************************************
ok: [default] => (item=/tmp/degoss.zaRypz2oGk)
ok: [default] => (item=/tmp/degoss.zaRypz2oGk/tests)
ok: [default] => (item=/tmp/degoss.zaRypz2oGk/bin)

TASK [degoss : install] ********************************************************
ok: [default] => (item=0.3.6)

TASK [degoss : link] ***********************************************************
ok: [default]

TASK [degoss : deploy test files] **********************************************
ok: [default] => (item=tests/goss.yml)

TASK [degoss : run tests] ******************************************************
ok: [default]

TASK [degoss : clean] **********************************************************
ok: [default]

TASK [degoss : report errors] **************************************************
fatal: [default]: FAILED! => {"changed": false, "msg": "Goss Tests Failed."}
    to retry, use: --limit @/Users/ccorbett/redacted/ansible-roles/common/vagrant.retry

If this points to an issue with goss itself, can you recommend a version to pin?

[naftulikay]

This is an issue kind of within Ansible, but ultimately something we could theoretically address.

What's happening here is that the Git repository here has tests/goss.yml as a test of the degoss role itself. If you use degoss with goss_file set to tests/goss.yml, it'll see the file in the role source and use that rather than the path you're trying to point to.

The solution seems to be to somehow ignore tests/ in Ansible Galaxy and to remove some defaults for goss_file and make it fail hard and fast when goss_file is undefined or not found.

[naftulikay]

Ansible Galaxy doesn't seem to have a way to ignore files, which sucks: https://github.com/ansible/galaxy/issues/78

[asciifaceman]

So it appears that if you override goss_file it either ignores it, or it uses the file local to the role itself. Then if you, for instance, override it with a full local path as an example, it uses that server side because

# run the tests
- name: run tests
  goss:
    executable: "{{ degoss_goss_bin }}"
    path: "{{ goss_file }}"
    cwd: "{{ degoss_test_root }}"
    format: "{{ goss_output_format }}"
    env_vars: "{{ goss_env_vars }}"
  failed_when: false
  register: goss_output
  tags: [format_goss_output]

...

TASK [degoss : debug goss] *****************************************************
ok: [default] => {
    "msg": {
        "changed": false,
        "failed": false,
        "failed_when_result": false,
        "goss_failed": true,
        "msg": "Goss test file /tmp/degoss.l6T2iLbttY/tests/~/ansible-roles/ansible.thing-common/tests/goss.yml is not readable."
    }
}

[asciifaceman]

- name: Run goss tests
  hosts: all
  roles:
    - { role: degoss, degoss_clean: false, degoss_debug: true, goss_file: tests/mygoss.yml}

TASK [degoss : debug goss] *****************************************************
ok: [default] => {
    "msg": {
        "changed": false,
        "failed": false,
        "failed_when_result": false,
        "goss_failed": true,
        "msg": "Goss test file /tmp/degoss.MLF77g97e5/tests/tests/mygoss.yml is not readable."
    }
}

[asciifaceman]

To prove it is breaking here, I cp'd my goss test to my roles root dir and ran with just goss_file: mygoss.yml

TASK [degoss : debug goss] *****************************************************
ok: [default] => {
    "msg": {
        "changed": false,
        "failed": false,
        "failed_when_result": false,
        "module_stderr": "Shared connection to 127.0.0.1 closed.\r\n",
        "module_stdout": "{}\r\n\r\n{\"stdout\": \"......\\n\\nTotal Duration: 0.014s\\nCount: 6, Failed: 0, Skipped: 0\\n\", \"changed\": false, \"failed\": false, \"goss_failed\": false, \"stderr\": \"\", \"rc\": 0, \"invocation\": {\"module_args\": {\"path\": \"mygoss.yml\", \"executable\": \"/tmp/degoss.kbqGFh1Y9s/bin/goss\", \"cwd\": \"/tmp/degoss.kbqGFh1Y9s/tests\", \"env_vars\": {}, \"format\": \"rspecish\"}}}\r\n",
        "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
        "rc": 0
    }
}

TASK [degoss : clean] **********************************************************
skipping: [default]

TASK [degoss : report errors] **************************************************
skipping: [default]

TASK [degoss : report module errors] *******************************************
skipping: [default]

potentially a new can of worms?

[asciifaceman]

testing

[asciifaceman]

In my use case:

goss_file: tests/goss/goss.yml

usage:

- name: Run goss tests
  hosts: all
  roles:
    - { role: degoss, degoss_clean: false, degoss_debug: true, goss_file: tests/goss/goss.yml}

TASK [degoss : run tests] ******************************************************
ok: [default]

[asciifaceman]

A+ @naftulikay thanks for being awesome