Failing cases not caught still, debug does not work

[asciifaceman]

17:36 $ ansible --version ansible 2.7.0

example role defaults:

---
sslRootDir: /etc/ssl
ssls:
  - name: star.domain.com
    bundle: bundle.crt
    key: some.key
    owner: root

example role main.yml:

---
- name: Make sure SSL directory exists
  become: true
  file:
    path: "{{ sslRootDir }}"
    state: directory
    owner: root
    group: root
    mode: 0755

- name: Make individual certificate directories
  become: true
  file:
    path: "{{ sslRootDir }}/{{ item.name }}"
    state: directory
    owner: root
    group: root
    mode: 0755
  loop: "{{ ssls }}"

- name: Install certificate bundles
  become: true
  copy:
    owner: "{{ item.owner }}"
    group: "{{ item.owner }}"
    src: "{{ item.name }}/{{ item.bundle }}"
    dest: "{{ sslRootDir }}/{{ item.name }}/{{ item.bundle }}"
    backup: yes
  loop: "{{ ssls }}"

- name: Install signing keys
  become: true
  copy:
    owner: "{{ item.owner }}"
    group: "{{ item.owner }}"
    src: "{{ item.name }}/{{ item.key }}"
    dest: "{{ sslRootDir }}/{{ item.name }}/{{ item.key }}"
    backup: yes
  loop: "{{ ssls }}"

just throw some files with the names bundle.crt and some.key in files/ for testing

goss.yml:

file:
  /etc/ssl:
    exists: true
    mode: "0755"
    owner: root
    group: root
    filetype: directory
  {{range .Env.ssls}}
  /etc/ssl/{{.name}}:
    exists: false
  /etc/ssl/{{.name}}/{{.bundle}}:
    exists: false
  /etc/ssl/{{.name}}/{{.key}}:
    exists: false
  {{end}}

These three tests fail on command line as expected:

File: /etc/ssl/star.domain.com/some.key: exists:
Expected
    <bool>: true
to equal
    <bool>: false

But degoss glosses right over:

TASK [degoss : deploy test files] **********************************************
ok: [default] => (item=tests/goss/goss.yml)

TASK [degoss : run tests] ******************************************************
ok: [default]

TASK [degoss : clean] **********************************************************
skipping: [default]

TASK [degoss : report errors] **************************************************
skipping: [default]

TASK [degoss : report module errors] *******************************************
skipping: [default]

PLAY RECAP *********************************************************************
default                    : ok=20   changed=6    unreachable=0    failed=0

Called:

- name: Run goss tests
  hosts: all
  roles:
    - role: degoss
      degoss_debug: true
      degoss_clean: false
      goss_file: tests/goss/goss.yml
      goss_env_vars:
        ssls: ssls

[asciifaceman]

Note: testing remote-side I'm quickly throwing defaults/main.yml into a vars.yml in the tmp dir, changing .Env to .Vars and running bin/goss --vars vars.yml -g tests/goss.yml validate --f documentation -o verbose

[asciifaceman]

Tested with

file:
  /derp/shit:
    exists: true
  /etc/passwd:
    exists: false

TASK [degoss : run tests] ******************************************************
ok: [default]

TASK [degoss : clean] **********************************************************
skipping: [default]

TASK [degoss : report errors] **************************************************
skipping: [default]

TASK [degoss : report module errors] *******************************************
skipping: [default]

PLAY RECAP *********************************************************************
default                    : ok=20   changed=6    unreachable=0    failed=0

[asciifaceman]

tasks/main.yml

Starting at line 67

# run the tests
- name: run tests
  goss:
    executable: "{{ degoss_goss_bin }}"
    path: "./{{ goss_file_basename }}"
    cwd: "{{ degoss_test_root }}"
    format: "{{ goss_output_format }}"
    env_vars: "{{ goss_env_vars }}"
  failed_when: false
  register: goss_output
  tags: [format_goss_output]

- name: debug goss
  debug:
    msg: "{{ goss_output.get('goss_failed') }}"

Result:

TASK [degoss : run tests] ******************************************************
ok: [default]

TASK [degoss : debug goss] *****************************************************
ok: [default] => {
    "msg": ""
}

[asciifaceman]

- name: get goss status
  set_fact:
    goss_stdout: "{{ goss_output.get('module_stdout') }}"

- name: debug goss 2
  debug:
    msg: "{{ goss_stdout }}"

TASK [degoss : debug goss 2] ***************************************************
ok: [default] => {
    "msg": "{}\r\n\r\n{\"stdout\": \"F....\\n\\nFailures/Skipped:\\n\\nFile: /etc/ssl: exists:\\nExpected\\n    <bool>: true\\nto equal\\n    <bool>: false\\n\\nTotal Duration: 0.000s\\nCount: 5, Failed: 1, Skipped: 0\\n\", \"failed\": true, \"goss_failed\": true, \"stderr\": \"\", \"rc\": 1, \"invocation\": {\"module_args\": {\"path\": \"./goss.yml\", \"executable\": \"/tmp/degoss.plyD9Jlk52/bin/goss\", \"cwd\": \"/tmp/degoss.plyD9Jlk52/tests\", \"env_vars\": {}, \"format\": \"rspecish\"}}, \"msg\": \"Goss Tests Failed.\"}\r\n"
}

But I cannot parse anything deeper than this, it appears to be too garbled

[asciifaceman]

It appears the erroneous data is worming its way into module_stdout, specifically {}\r\n\r\n prefixing the real payload.

- name: get goss status
  set_fact:
    goss_stdout: "{{ goss_output.get('module_stdout').lstrip('{}\r\n\r\n') | from_json }}"

# our output callback plugin will catch the tag of this and format output accordingly
- name: report errors
  fail:
    msg:
      message: Goss Tests Failed.
      rc: "{{ goss_output.rc }}"
      tmp_root: "{{ degoss_tmp_root }}"
      version: "{{ goss_real_version }}"
      debug: "{{ goss_stdout }}"
  when: goss_stdout.get('goss_failed') | bool

This appears to make it operate as expected, but of course this is a dirty hack around some erroneous extra payload that shouldn't be there

[naftulikay]

Please retest against the latest 3.0.0 and re-open if the issue is still present. #36 saw a complete and total refactor of everything, extensive unit and integration testing, etc.