logging does not work when running jsnapy with ansible

avocadonetwork1 commented 2 years ago

Description of Issue/Question

Hi, I have integrated jsnapy with ansible so that I can use ansible playbook to run jsnapy. However the info and err from my jsnapy tests are not outputing when I run the ansible playbook. As the documentation suggests, I have tried both of the configs below on ansible.cfg but still no output callback_enabled = juniper.device.jsnapy callback_whitelist = juniper.device.jsnapy

Setup

The following files are created

snapcheck_playbook.yml (ansible playbook) ,
jsnapy_tests.yml (which tests I am running) and
test_snmp.yml (test itself)

I am adding the files below

` snapcheck_playbook.yml

name: "Ansible Jsnapy Snapcheck" hosts: all strategy: free connection: local gather_facts: false

tasks:
- name: Run Jsnapy check juniper.device.jsnapy: action: snapcheck config_file: jsnapy_tests.yml logfile: jsnapy_ansible.log register: result
- name: Run Jsnapy check assert: that:
  - "result.passPercentage == 100"

jsnapy_tests.yml

tests:

test_snmp.yml

test_snmp.yml

Test SNMP config:

rpc: get-configuration
item: xpath: /configuration/snmp tests:
- exists: community[name='mycommunity'] info: SNMP community ''mycommunity' is configured. err: SNMP community ''mycommunity' is not configured! `
  Steps to Reproduce Issue
  
  (Include debug logs if possible and relevant. Error trace would be helpful too) To run the playbook: $ ansible-playbook snapcheck_playbook.yml -l mydevice

PLAY [Ansible Jsnapy Snapcheck] ***

TASK [Run Jsnapy check] *** ok: [mydevice]

TASK [Run Jsnapy check] *** fatal: [mydevice]: FAILED! => { "assertion": "result.passPercentage == 100", "changed": false, "evaluated_to": false, "msg": "Assertion failed" }

PLAY RECAP **** mydevice : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0

As you can see from the above, the tests failed as expected (because the snmp community is not set at my router yet. However the err message is not displayed on the output. I can see the errors on the jsnapy.log but not on the output from the command

Versions Report

(Provided by running jsnapy --version. Please also mention python version.) $ jsnapy --version JSNAPy version: 1.3.6

$ ansible --version ansible [core 2.11.1]

$ python --version Python 3.9.5

chidanandpujar commented 3 months ago

issue is reproducible with the following information


cat snapcheck_playbook.yml 
---
- name: Test juniper.device.jsnapy module
  hosts: all
  gather_facts: false
  tasks:
##################################################
####    TEST 2                                  ##
##################################################
    - name: "TEST 2 - Execute SNAPCHECK "
      juniper.device.jsnapy:
        config_file: jsnapy_tests.yml
        action: snapcheck
      register: result 

    - name: Verify JSNAPy tests passed
      assert:
        that:
          - "result.passPercentage == 100"

cat jsnapy_tests.yml 
tests:
  test_snmp.yml

 cat test_snmp.yml 
tests_include:
  - test_snmp_config 

test_snmp_config:
  - rpc: get-configuration
  - iterate:
      xpath: /configuration/snmp
      tests:
        - exists: community[name='mycommunity'] 
          info: SNMP community ''mycommunity' is configured. 
          err: SNMP community ''mycommunity' is not configured!

jsnapy --snapcheck -f jsnapy_tests_with_host.yml
Connecting to device x.x.x.x ................
Taking snapshot of RPC: get-configuration
**************************** Device: x.x.x.x ****************************
Tests Included: test_snmp_config 
*************************RPC is get-configuration*************************
ERROR!! Nodes are not present in given <Xpath> : </configuration/snmp>
FAIL | All "community[name='mycommunity']" do not exists at xpath "/configuration/snmp" [ 0 value matched / 1 value failed ]
------------------------------- Final Result!! -------------------------------
test_snmp_config : Failed
Total No of tests passed: 0
Total No of tests failed: 1 
Overall Tests failed!!! 

ansible-playbook snapcheck_playbook.yml 

PLAY [Test juniper.device.jsnapy module] **********************************************************************************************************************************

TASK [TEST 2 - Execute SNAPCHECK] *****************************************************************************************************************************************
ok: [local_connection_testcases_mode_ssh]

TASK [Verify JSNAPy tests passed] *****************************************************************************************************************************************
fatal: [local_connection_testcases_mode_ssh]: FAILED! => changed=false 
  assertion: result.passPercentage == 100
  evaluated_to: false
  msg: Assertion failed

PLAY RECAP ****************************************************************************************************************************************************************
local_connection_testcases_mode_ssh : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Thanks Chidanand

chidanandpujar commented 1 month ago

Pass logs:


##################################################
####    TEST JSNAPY PRE                         ##
##################################################

    - name: "Check after PRE - POST check"
      juniper.device.jsnapy:
        test_files: test_snmp.yml
        action: "snapcheck"
        dest_dir: "{{ backup_dir }}"  
      register: test_check
      tags: 
        - jsnapy

    - debug: var=test_check

TASK [Check after PRE - POST check] ***************************************************************************************************************************************
ok: [x.x.x.x]

TASK [debug] **************************************************************************************************************************************************************
ok: [x.x.x.x] => {
    "test_check": {
        "changed": false,
        "failed": false,
        "get-configuration": [
            {
                "count": {
                    "fail": 1,
                    "pass": 0
                },
                "err": "SNMP community ''mycommunity' is not configured!",
                "failed": [
                    {
                        "actual_node_value": null,
                        "id": {},
                        "post": {},
                        "pre": {},
                        "xpath_error": true
                    }
                ],
                "node_name": "community[name='mycommunity']",
                "passed": [],
                "result": false,
                "test_name": "test_snmp_config",
                "testoperation": "exists",
                "xpath": "/configuration/snmp"
            }
        ]
    }
}

PLAY RECAP ****************************************************************************************************************************************************************
x.x.x.x              : ok=8    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

cat backup_dir/x.x.x.x_test_snmp_config_False.text 
[{'xpath': '/configuration/snmp', 'testoperation': 'exists', 'passed': [], 'failed': [{'id': {}, 'pre': {}, 'post': {}, 'actual_node_value': None, 'xpath_error': True}], 'test_name': 'test_snmp_config', 'node_name': "community[name='mycommunity']", 'err': "SNMP community ''mycommunity' is not configured!", 'result': False, 'count': {'pass': 0, 'fail': 1}}]

Thanks Chidanand

chidanandpujar commented 1 month ago

operator exists now reports info/err messages as part of fix #415 and also failed tests can be copied to dest_dir and it is supported via https://github.com/Juniper/ansible-junos-stdlib/pull/678 callback issue will be handled in the ansible-junos-stdlib via issue https://github.com/Juniper/ansible-junos-stdlib/issues/599

Juniper / jsnapy