Feat(cv_device_v3): add configuration validation support

[noredistribution]

Change Summary

Adding support for configlet validation against devices

Related Issue(s)

Fixes #552

Component(s) name

arista.cvp.cv_device_v3

Proposed changes

added a new state called validate that leverages validate_config_for_device and validate_configlets_for_device functions from cvprac

How to test

Unit test

pytest tests/unit/test_cv_device_tools.py -s

Molecule test

Update ansible_httpapi_host, ansible_host, ansible_user, ansible_password in ansible_collection/arista/cvp/examples/inventory.yml and run molecule converge -s cv_device_v3 Log file: device_validate_config_LOG.log

End to End test

• Follow the instruction on Readme to setup testing environment.
• Run the following playbooks (found on your ATD under persist/PR_testing/cv_device_v3) and verify the results as shown
• NOTE: Please make sure to test each playbook with all available options for validate_mode variable. validate_mode=stop_on_error validate_mode=stop_on_warning validate_mode=ignore

1. device_validate_config_valid.yaml with validate_mode=stop_on_warning or validate_mode=stop_on_error or validate_mode=ignore

$ ansible-playbook cv_device_v3/device_validate_config_valid.yaml -i inventory.yaml

TASK [print result for CloudVision] ****************************************************************************************************************
ok: [CloudVision] => 
  msg:
    changed: true
    configlets_validated:
      changed: true
      configlets_validated_count: 1
      configlets_validated_list:
      - validate_valid_on_leaf1_validated
      diff: {}
      errors: []
      success: true
      taskIds: []
      warnings: []
    failed: false
    success: true
    taskIds: []

PLAY RECAP *****************************************************************************************************************************************
CloudVision                : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

2. device_validate_config_warning.yaml with validate_mode=stop_on_warning

$ ansible-playbook cv_device_v3/device_validate_config_warning.yaml -i inventory.yaml

TASK [Validate configurations] *********************************************************************************************************************
fatal: [CloudVision]: FAILED! => changed=false 
  configlets_validated:
    configlets_validated_count: 1
    configlets_validated_list:
    - validate_warning_on_leaf1_validated
    diff: {}
    errors: []
    success: true
    taskIds: []
    warnings:
    - device: leaf1
      warnings:
      - '! portfast should only be enabled on ports connected to a single host. Connecting hubs, concentrators, switches, bridges, etc. to this interface when portfast is enabled can cause temporary bridging loops. Use with CAUTION. at line 2'
  msg: Encountered 1 warnings during validation. Refer to 'configlets_validated' for more details.

PLAY RECAP *****************************************************************************************************************************************
CloudVision                : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

3. device_validate_config_warning.yaml with validate_mode=stop_on_error or validate_mode=ignore

$ ansible-playbook cv_device_v3/device_validate_config_warning.yaml -i inventory.yaml

TASK [print result for CloudVision] ****************************************************************************************************************
ok: [CloudVision] => 
  msg:
    changed: true
    configlets_validated:
      changed: true
      configlets_validated_count: 1
      configlets_validated_list:
      - validate_warning_on_leaf1_validated
      diff: {}
      errors: []
      success: true
      taskIds: []
      warnings:
      - device: leaf1
        warnings:
        - '! portfast should only be enabled on ports connected to a single host. Connecting hubs, concentrators, switches, bridges, etc. to this interface when portfast is enabled can cause temporary bridging loops. Use with CAUTION. at line 2'
    failed: false
    success: true
    taskIds: []

PLAY RECAP *****************************************************************************************************************************************
CloudVision                : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

4. device_validate_config_error.yaml with validate_mode=stop_on_error or validate_mode=stop_on_warning

$ ansible-playbook cv_device_v3/device_validate_config_error.yaml -i inventory.yaml

TASK [Validate configurations] *********************************************************************************************************************
fatal: [CloudVision]: FAILED! => changed=false 
  configlets_validated:
    configlets_validated_count: 1
    configlets_validated_list:
    - validate_error_on_leaf1_validated
    diff: {}
    errors:
    - device: leaf1
      errors:
      - error: '> ruter bgp 1111% Invalid input (at token 0: ''ruter'') at line 1'
        lineNo: ' 1'
      - error: '>    neighbor 1.1.1.1 remote-bs 111% Invalid input (at token 1: ''1.1.1.1'') at line 2'
        lineNo: ' 2'
    success: true
    taskIds: []
    warnings: []
  msg: Encountered 1 errors during validation. Refer to 'configlets_validated' for more details.

PLAY RECAP *****************************************************************************************************************************************
CloudVision                : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

5. device_validate_config_error.yaml with validate_mode=ignore

$ ansible-playbook cv_device_v3/device_validate_config_error.yaml -i inventory.yaml

TASK [print result for CloudVision] ****************************************************************************************************************
ok: [CloudVision] => 
  msg:
    changed: true
    configlets_validated:
      changed: true
      configlets_validated_count: 1
      configlets_validated_list:
      - validate_error_on_leaf1_validated
      diff: {}
      errors:
      - device: leaf1
        errors:
        - error: '> ruter bgp 1111% Invalid input (at token 0: ''ruter'') at line 1'
          lineNo: ' 1'
        - error: '>    neighbor 1.1.1.1 remote-bs 111% Invalid input (at token 1: ''1.1.1.1'') at line 2'
          lineNo: ' 2'
      success: true
      taskIds: []
      warnings: []
    failed: false
    success: true
    taskIds: []

PLAY RECAP *****************************************************************************************************************************************
CloudVision                : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

6. device_validate_config_multiple.yaml with validate_mode=stop_on_error or validate_mode=stop_on_warning

$ ansible-playbook cv_device_v3/device_validate_config_multiple.yaml -i inventory.yaml

TASK [Validate configurations] *********************************************************************************************************************
fatal: [CloudVision]: FAILED! => changed=false 
  configlets_validated:
    configlets_validated_count: 3
    configlets_validated_list:
    - validate_error_on_leaf1_validated
    - validate_valid_on_leaf1_validated
    - validate_warning_on_leaf1_validated
    diff: {}
    errors:
    - device: leaf1
      errors:
      - error: '> ruter bgp 1111% Invalid input (at token 0: ''ruter'') at line 1'
        lineNo: ' 1'
      - error: '>    neighbor 1.1.1.1 remote-bs 111% Invalid input (at token 1: ''1.1.1.1'') at line 2'
        lineNo: ' 2'
    success: true
    taskIds: []
    warnings:
    - device: leaf1
      warnings:
      - '! portfast should only be enabled on ports connected to a single host. Connecting hubs, concentrators, switches, bridges, etc. to this interface when portfast is enabled can cause temporary bridging loops. Use with CAUTION. at line 2'
  msg: Encountered 1 errors during validation. Refer to 'configlets_validated' for more details.

PLAY RECAP *****************************************************************************************************************************************
CloudVision                : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

7. device_validate_config_multiple.yaml with validate_mode=ignore

$ ansible-playbook cv_device_v3/device_validate_config_multiple.yaml -i inventory.yaml

TASK [print result for CloudVision] ****************************************************************************************************************
ok: [CloudVision] => 
  msg:
    changed: true
    configlets_validated:
      changed: true
      configlets_validated_count: 3
      configlets_validated_list:
      - validate_error_on_leaf1_validated
      - validate_valid_on_leaf1_validated
      - validate_warning_on_leaf1_validated
      diff: {}
      errors:
      - device: leaf1
        errors:
        - error: '> ruter bgp 1111% Invalid input (at token 0: ''ruter'') at line 1'
          lineNo: ' 1'
        - error: '>    neighbor 1.1.1.1 remote-bs 111% Invalid input (at token 1: ''1.1.1.1'') at line 2'
          lineNo: ' 2'
      success: true
      taskIds: []
      warnings:
      - device: leaf1
        warnings:
        - '! portfast should only be enabled on ports connected to a single host. Connecting hubs, concentrators, switches, bridges, etc. to this interface when portfast is enabled can cause temporary bridging loops. Use with CAUTION. at line 2'
    failed: false
    success: true
    taskIds: []

PLAY RECAP *****************************************************************************************************************************************
CloudVision                : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0      

Checklist

User Checklist

• N/A

Repository Checklist

• [ x] My code has been rebased from devel before I start
• [ x] I have read the CONTRIBUTING document.
• [x] My change requires a change to the documentation and documentation have been updated accordingly. (check the box if not applicable)
• [ ] I have updated molecule CI testing accordingly. (check the box if not applicable)

[noredistribution]

What is the difference between stop_on_warning and stop_on_error ? They both seem to behave the same way for your example playbook?

The difference is that one stops if you have a warning, the other stops if you have an error, it just happens to be that the configlet has both of them in this case because Ethernet 10 doesn't exist on my veos-lab, which is a warning, but also ruter is not a valid word, which will be a warning. You can check the logic at the end of the validate_config function from line 1873; I tried to mimic what the UI does today

[chetryan]

Opinion/Question : Is it worth moving errors and warnings to a different field than msg? If we want to use the warnings/errors it might be easier to do if we don't have to parse msg again.

[github-actions[bot]]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions[bot]]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[mstreet14]

I went through each of the tests cases and the playbooks worked as expected for each of the conditions outlined.

[rwedmonds]

PR Test Results

PR #	Test	Command	Output	Tests Successful?
553	device_validate_config_error.yaml	ansible-playbook cv_device_v3/device_validate_config_error.yaml -i inventory.yaml validate_mode: stop_on_error	PLAY [Device Config Validation in Cloudvision] **** TASK [Push configlet] **** changed: [CloudVision] TASK [Validate configurations] **** fatal: [CloudVision]: FAILED! => changed=false configlets_validated: configlets_validated_count: 1 configlets_validated_list: - validate_error_on_leaf1_validated diff: {} errors: - device: leaf1 errors: - error: '> ruter bgp 1111% Invalid input (at token 0: ''ruter'') at line 1' lineNo: ' 1' - error: '> neighbor 1.1.1.1 remote-bs 111% Invalid input (at token 1: ''1.1.1.1'') at line 2' lineNo: ' 2' success: true taskIds: [] warnings: [] msg: Encountered 1 errors during validation. Refer to 'configlets_validated' for more details. PLAY RECAP **** CloudVision : ok=1 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0	:white_check_mark:
	device_validate_config_error.yaml	ansible-playbook cv_device_v3/device_validate_config_error.yaml -i inventory.yaml validate_mode: stop_on_warning	PLAY [Device Config Validation in Cloudvision] ** TASK [Push configlet] ** ok: [CloudVision] TASK [Validate configurations] ** fatal: [CloudVision]: FAILED! => changed=false configlets_validated: configlets_validated_count: 1 configlets_validated_list: - validate_error_on_leaf1_validated diff: {} errors: - device: leaf1 errors: - error: '> ruter bgp 1111% Invalid input (at token 0: ''ruter'') at line 1' lineNo: ' 1' - error: '> neighbor 1.1.1.1 remote-bs 111% Invalid input (at token 1: ''1.1.1.1'') at line 2' lineNo: ' 2' success: true taskIds: [] warnings: [] msg: Encountered 1 errors during validation. Refer to 'configlets_validated' for more details. PLAY RECAP ** CloudVision : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0	:white_check_mark:
	device_validate_config_error.yaml	ansible-playbook cv_device_v3/device_validate_config_error.yaml -i inventory.yaml validate_mode: ignore	PLAY [Device Config Validation in Cloudvision] ** TASK [Push configlet] ** ok: [CloudVision] TASK [Validate configurations] ** changed: [CloudVision] TASK [print result for CloudVision] ** ok: [CloudVision] => msg: changed: true configlets_validated: changed: true configlets_validated_count: 1 configlets_validated_list: - validate_error_on_leaf1_validated diff: {} errors: - device: leaf1 errors: - error: '> ruter bgp 1111% Invalid input (at token 0: ''ruter'') at line 1' lineNo: ' 1' - error: '> neighbor 1.1.1.1 remote-bs 111% Invalid input (at token 1: ''1.1.1.1'') at line 2' lineNo: ' 2' success: true taskIds: [] warnings: [] failed: false success: true taskIds: [] PLAY RECAP ** CloudVision : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0	:white_check_mark:
	device_validate_config_multiple.yaml	ansible-playbook cv_device_v3/device_validate_config_multiple.yaml -i inventory.yaml validate_mode: stop_on_error	PLAY [Device Config Validation in Cloudvision] **** TASK [Push configlet] **** changed: [CloudVision] TASK [Validate configurations] **** fatal: [CloudVision]: FAILED! => changed=false configlets_validated: configlets_validated_count: 3 configlets_validated_list: - validate_error_on_leaf1_validated - validate_valid_on_leaf1_validated - validate_warning_on_leaf1_validated diff: {} errors: - device: leaf1 errors: - error: '> ruter bgp 1111% Invalid input (at token 0: ''ruter'') at line 1' lineNo: ' 1' - error: '> neighbor 1.1.1.1 remote-bs 111% Invalid input (at token 1: ''1.1.1.1'') at line 2' lineNo: ' 2' success: true taskIds: [] warnings: - device: leaf1 warnings: - '! portfast should only be enabled on ports connected to a single host. Connecting hubs, concentrators, switches, bridges, etc. to this interface when portfast is enabled can cause temporary bridging loops. Use with CAUTION. at line 2' msg: Encountered 1 errors during validation. Refer to 'configlets_validated' for more details. PLAY RECAP **** CloudVision : ok=1 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0	:white_check_mark:
	device_validate_config_multiple.yaml	ansible-playbook cv_device_v3/device_validate_config_multiple.yaml -i inventory.yaml validate_mode: stop_on_warning	PLAY [Device Config Validation in Cloudvision] TASK [Push configlet] ok: [CloudVision] TASK [Validate configurations] fatal: [CloudVision]: FAILED! => changed=false configlets_validated: configlets_validated_count: 3 configlets_validated_list: - validate_error_on_leaf1_validated - validate_valid_on_leaf1_validated - validate_warning_on_leaf1_validated diff: {} errors: - device: leaf1 errors: - error: '> ruter bgp 1111% Invalid input (at token 0: ''ruter'') at line 1' lineNo: ' 1' - error: '> neighbor 1.1.1.1 remote-bs 111% Invalid input (at token 1: ''1.1.1.1'') at line 2' lineNo: ' 2' success: true taskIds: [] warnings: - device: leaf1 warnings: - '! portfast should only be enabled on ports connected to a single host. Connecting hubs, concentrators, switches, bridges, etc. to this interface when portfast is enabled can cause temporary bridging loops. Use with CAUTION. at line 2' msg: Encountered 1 errors during validation. Refer to 'configlets_validated' for more details. PLAY RECAP CloudVision : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0	:white_check_mark:
	device_validate_config_multiple.yaml	ansible-playbook cv_device_v3/device_validate_config_multiple.yaml -i inventory.yaml validate_mode: ignore	PLAY [Device Config Validation in Cloudvision] TASK [Push configlet] ok: [CloudVision] TASK [Validate configurations] changed: [CloudVision] TASK [print result for CloudVision] ok: [CloudVision] => msg: changed: true configlets_validated: changed: true configlets_validated_count: 3 configlets_validated_list: - validate_error_on_leaf1_validated - validate_valid_on_leaf1_validated - validate_warning_on_leaf1_validated diff: {} errors: - device: leaf1 errors: - error: '> ruter bgp 1111% Invalid input (at token 0: ''ruter'') at line 1' lineNo: ' 1' - error: '> neighbor 1.1.1.1 remote-bs 111% Invalid input (at token 1: ''1.1.1.1'') at line 2' lineNo: ' 2' success: true taskIds: [] warnings: - device: leaf1 warnings: - '! portfast should only be enabled on ports connected to a single host. Connecting hubs, concentrators, switches, bridges, etc. to this interface when portfast is enabled can cause temporary bridging loops. Use with CAUTION. at line 2' failed: false success: true taskIds: [] PLAY RECAP *** CloudVision : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0	:white_check_mark:
	device_validate_config_valid.yaml	ansible-playbook cv_device_v3/device_validate_config_valid.yaml -i inventory.yaml validate_mode: stop_on_error	PLAY [Device Config Validation in Cloudvision] **** TASK [Push configlet] *** ok: [CloudVision] TASK [Validate configurations]**** changed: [CloudVision] TASK [print result for CloudVision] * ok: [CloudVision] => msg: changed: true configlets_validated: changed: true configlets_validated_count: 1 configlets_validated_list: - validate_valid_on_leaf1_validated diff: {} errors: [] success: true taskIds: [] warnings: [] failed: false success: true taskIds: [] PLAY RECAP **** CloudVision : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0	:white_check_mark:
	device_validate_config_valid.yaml	ansible-playbook cv_device_v3/device_validate_config_valid.yaml -i inventory.yaml validate_mode: stop_on_warning	PLAY [Device Config Validation in Cloudvision] **** TASK [Push configlet] **** ok: [CloudVision] TASK [Validate configurations] **** changed: [CloudVision] TASK [print result for CloudVision] **** ok: [CloudVision] => msg: changed: true configlets_validated: changed: true configlets_validated_count: 1 configlets_validated_list: - validate_valid_on_leaf1_validated diff: {} errors: [] success: true taskIds: [] warnings: [] failed: false success: true taskIds: [] PLAY RECAP **** CloudVision : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0	:white_check_mark:
	device_validate_config_valid.yaml	ansible-playbook cv_device_v3/device_validate_config_valid.yaml -i inventory.yaml validate_mode: ignore	PLAY [Device Config Validation in Cloudvision] **** TASK [Push configlet] **** ok: [CloudVision] TASK [Validate configurations] **** changed: [CloudVision] TASK [print result for CloudVision] **** ok: [CloudVision] => msg: changed: true configlets_validated: changed: true configlets_validated_count: 1 configlets_validated_list: - validate_valid_on_leaf1_validated diff: {} errors: [] success: true taskIds: [] warnings: [] failed: false success: true taskIds: [] PLAY RECAP **** CloudVision : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
	device_validate_config_warning.yaml	ansible-playbook cv_device_v3/device_validate_config_warning.yaml -i inventory.yaml validate_mode: stop_on_error	PLAY [Device Config Validation in Cloudvision] **** TASK [Push configlet] **** ok: [CloudVision] TASK [Validate configurations] **** changed: [CloudVision] TASK [print result for CloudVision] **** ok: [CloudVision] => msg: changed: true configlets_validated: changed: true configlets_validated_count: 1 configlets_validated_list: - validate_warning_on_leaf1_validated diff: {} errors: [] success: true taskIds: [] warnings: - device: leaf1 warnings: - '! portfast should only be enabled on ports connected to a single host. Connecting hubs, concentrators, switches, bridges, etc. to this interface when portfast is enabled can cause temporary bridging loops. Use with CAUTION. at line 2' failed: false success: true taskIds: [] PLAY RECAP **** CloudVision : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0	:white_check_mark:
	device_validate_config_warning.yaml	ansible-playbook cv_device_v3/device_validate_config_warning.yaml -i inventory.yaml validate_mode: stop_on_warning	PLAY [Device Config Validation in Cloudvision] **** TASK [Push configlet] **** ok: [CloudVision] TASK [Validate configurations] **** fatal: [CloudVision]: FAILED! => changed=false configlets_validated: configlets_validated_count: 1 configlets_validated_list: - validate_warning_on_leaf1_validated diff: {} errors: [] success: true taskIds: [] warnings: - device: leaf1 warnings: - '! portfast should only be enabled on ports connected to a single host. Connecting hubs, concentrators, switches, bridges, etc. to this interface when portfast is enabled can cause temporary bridging loops. Use with CAUTION. at line 2' msg: Encountered 1 warnings during validation. Refer to 'configlets_validated' for more details. PLAY RECAP **** CloudVision : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0	:white_check_mark:
	device_validate_config_warning.yaml	ansible-playbook cv_device_v3/device_validate_config_warning.yaml -i inventory.yaml validate_mode: ignore	PLAY [Device Config Validation in Cloudvision] **** TASK [Push configlet] **** ok: [CloudVision] TASK [Validate configurations] **** changed: [CloudVision] TASK [print result for CloudVision] **** ok: [CloudVision] => msg: changed: true configlets_validated: changed: true configlets_validated_count: 1 configlets_validated_list: - validate_warning_on_leaf1_validated diff: {} errors: [] success: true taskIds: [] warnings: - device: leaf1 warnings: - '! portfast should only be enabled on ports connected to a single host. Connecting hubs, concentrators, switches, bridges, etc. to this interface when portfast is enabled can cause temporary bridging loops. Use with CAUTION. at line 2' failed: false success: true taskIds: [] PLAY RECAP **** CloudVision : ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0	:white_check_mark:

[github-actions[bot]]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions[bot]]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[shenzhiq]

I tested all the items and there is no issue

[noredistribution]

tested the latest commit with the molecule test and all went well

[chetryan]

I feel that the molecule tests capture all the possible scenarios we want to test. All the molecule tests passed when I tested the latest commit. LGTM!

[github-actions[bot]]

This pull request has conflicts, please resolve those before we can evaluate the pull request.