napalm-automation / napalm

Network Automation and Programmability Abstraction Layer with Multivendor support
Apache License 2.0
2.26k stars 554 forks source link

Napalm validate inside list search for patterns instead of fullmatch #2029

Open Tristou27 opened 7 months ago

Tristou27 commented 7 months ago

Description of Issue/Question

When using the validate / compliance_report function, the list validation is not done properly. For a given "item" in a validation file, if this item is specified as a list, then the _compare_getter_list() method is called. This method used compare() for each element in the list and if the elements are strings, then the re.search() method is used.

The issue is by doing that, false positive can be found as the re.search() method only look for a partial match. For example, if we have a validation file checking for "Vlan5" interface in the interface_list item, and that the device sends "Vlan5554", it will be considered as a positive result even though it's not.

A quick "fix" I can think of is the use of re.fullmatch() instead of re.search(). Or introduce a specific syntax in the validation file to precise if we're talking about pattern or "complete strings".

Did you follow the steps from https://github.com/napalm-automation/napalm#faq

(Place an x between the square brackets where applicable)

Setup

napalm version

(Paste verbatim output from pip freeze | grep napalm between quotes below)

# ttr @ serverin ~/autotest on git:main x .venv39 [17:42:42] 
$ pip freeze | grep napalm
napalm==4.0.0

Network operating system version

(Paste verbatim output from show version - or equivalent - between quotes below)

N/A

Steps to Reproduce the Issue

  1. Build a validation file for get_facts, check if the "interface_list" attribute contains "Vlan5" entry.
  2. Configure a device where a interface called Vlan5553 is created
  3. Run the validation, it will be detected as positive even though Vlan5 interface is not created on the device.
mirceaulinic commented 7 months ago

Sounds sensible to me! @Tristou27

Tristou27 commented 7 months ago

Thank you for the approval! I'll try to work on a PR soon.