Hello,

when I try to declare a complex regex in the template like below, I've got an error on the execution of the script. I soon as take of variable regex it works fine

Vars declaration

<var>
check = r"((?<!range\s).+?)"
</var>

Error message raised :

Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/ttp/ttp.py", line 1161, in parse_template_XML template_ET = ET.XML(template_text) File "/usr/lib/python3.6/xml/etree/ElementTree.py", line 1314, in XML parser.feed(text) xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 2, column 14

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/user/template/test_macro.py", line 14, in parser = ttp(data=result, template="/home/user/template/check_macro.ttp") File "/usr/local/lib/python3.6/dist-packages/ttp/ttp.py", line 189, in init self.add_template(template=template) File "/usr/local/lib/python3.6/dist-packages/ttp/ttp.py", line 311, in add_template ttp_macro=ttp.get("_customfunctions", {}).get("macro", {}) File "/usr/local/lib/python3.6/dist-packages/ttp/ttp.py", line 845, in init self.load_template_xml(template_text) File "/usr/local/lib/python3.6/dist-packages/ttp/ttp.py", line 1186, in load_template_xml parse_template_XML(template_text) File "/usr/local/lib/python3.6/dist-packages/ttp/ttp.py", line 1171, in parse_template_XML "".format(template_text) File "/usr/lib/python3.6/xml/etree/ElementTree.py", line 1314, in XML parser.feed(text) xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 3, column 14

Python program for the testing purpose

from netmiko import ConnectHandler
from ttp import ttp
import json

check={'device_type': 'cisco_ios','host': "r1",'username': "user",'password': "password","secret":"secret",'verbose' : True,'fast_cli':True}

connect=ConnectHandler(**check)
connect.enable()
result=connect.send_command("show running partition access-list")
print(result)
parser = ttp(data=result, template="/home/scamelo/bypass_enable_nornir/template/check_macro.ttp")
parser.parse()
results = parser.result(format='json')[0]
print(results)

ttp template

<var>
check =r"((?<!range\s).+?)"
</var>
<macro>

def check_acl_rule(data):

    if data["action"]=="remark":
        data["remark_name"]=data["action_detail"]
        data.pop("action_detail")

    else:
        itervalue=data["action_detail"].split(" ")
        if len(itervalue)==1:
            data["src_ip"]=itervalue[0]
            data.pop("action_detail")

        elif len(itervalue)==2:
            if itervalue[1]=="log":
                data["src_ip"]=itervalue[0]
                data["log"]="log"
                data.pop("action_detail")
                return data
            else:
                data["src_ip"]=itervalue[0]
                data["src_wildcard"]=itervalue[1]
                data.pop("action_detail")
                return data
        else:

            data["protocol"]=itervalue[0]
            itervalue.pop(0)
            newparse=data["action_detail"].replace(data["protocol"],"")
            print(itervalue)
            print(newparse)
            """
            for i in range(1, len(itervalue)-1):
                if re.match(r"host",itervalue[i]) is not None:
                    host_check=True
                    any_check=False
                elif re.match(r"host",itervalue[i]) is not None:
            data["toparse"]=itervalue
            """
    return data     

</macro>

<group name="ip.{{type}}.{{accesslist}}*">
ip access-list {{type}} {{accesslist|WORD|_start_}}
 <group name="{{rule_num}}*" macro="check_acl_rule">
 {{rule_num}} {{action}} {{action_detail|_line_}}
ip{{_end_}}
</group>
</group>

Could you please share sample of data with as many varieties as possible and desired structure you want to produce. Also seems that you parsing ACLs, have you had a look at capirca module?

Regarding xml.etree errors, its a restriction of XML, cannot use < > & characters in tag data - https://www.w3schools.com/xml/xml_syntax.asp - scroll to "Entity References" section.

Can think of two workarounds

for variables can use include attribute to load python variables from separate text file
can add template variables to TTP parser object on instantiation using vars attribute

Hello, Yes I'm parsing an ACL from cisco ios equipments. I don't think capirca will fit what I'm looking for.

Below an example of ACL that I'm trying to inspect.

ACL EXAMPLE

!
ip access-list standard 42
 10 remark machine_A
 10 permit 192.168.200.162
 20 remark machine_B
 20 permit 192.168.200.149
 30 deny   any log
ip access-list standard 98
 10 permit 10.10.10.1
 20 remark toto
 20 permit 30.30.30.1
 30 permit 30.30.30.0 0.0.0.255
ip access-list standard 99
 10 permit 10.20.30.40 log
 20 permit 20.30.40.1 log
 30 remark DEVICE - SNMP RW
 30 permit 50.50.50.128 0.0.0.127
 40 permit 60.60.60.64 0.0.0.63
ip access-list extended 199
 10 remark COLLECTOR - SNMP
 10 permit ip 70.70.70.0 0.0.0.255 any
 20 remark RETURN - Back
 20 permit ip 80.80.80.0 0.0.0.127 any
 30 remark VISUALIZE
 30 permit ip host 90.90.90.138 any
!

Strutuctured Data Expected

[
    [
        {
            "ip": {
                "extended": {
                    "199": [
                        {
                            "10": [
                                {
                                    "action": "remark",
                                    "remark_name": "COLLECTOR - SNMP"
                                },
                                {
                                    "action": "permit",
                                    "dest_any": "any",
                                    "protocol": "ip",
                                    "src_ntw": "70.70.70.0",
                                    "src_wildcard": "0.0.0.255"
                                }
                            ],
                            "20": [
                                {
                                    "action": "remark",
                                    "remark_name": "RETURN - Back"
                                },
                                {
                                    "action": "permit",
                                    "dest_any": "any",
                                    "protocol": "ip",
                                    "src_ntw": "80.80.80.0",
                                    "src_wildcard": "0.0.0.127",                            
                                }
                            ],
                            "30": [
                                {
                                    "action": "remark",
                                    "remark_name": "VISUALIZE"
                                },
                                {
                                    "action": "permit",
                                    "dest_any": "any",
                                    "protocol": "ip",
                                    "src_host": "90.90.90.138",
                                }
                            ],
                "standard": {
                    "42": [
                        {
                            "10": [
                                {
                                    "action": "remark",
                                    "remark_name": "machine_A"
                                },
                                {
                                    "action": "permit",
                                    "src_host": "192.168.200.162"
                                }
                            "20": [
                                {
                                    "action": "remark",
                                    "remark_name": "machine_B"
                                },
                                {
                                    "action": "permit",
                                    "src_host": "192.168.200.149"
                                }
                            ],
                            "30": [
                                {
                                    "action": "deny",
                                    "src_any": "any,
                                    "log": "log"
                                }
                            ],
                            ],
                        }
                    ],
                    "98": [
                        {
                            "10": [
                                {
                                    "action": "permit",
                                    "src_host": "10.10.10.1"
                                }
                            ],
                                "20": [
                                {
                                    "action": "remark",
                                    "remark_name": "toto"
                                },
                                {
                                    "action": "permit",
                                    "src_host": "30.30.30.1"
                                }
                            ],
                            "30": [
                                {
                                    "action": "permit",
                                    "src_ntw": "30.30.30.0",
                                    "src_wildcard":"0.0.0.255"
                                }
                            ],
                   "99": [
                        {
                            "10": [
                                {
                                    "action": "permit",
                                    "log": "log",
                                    "src_host": "10.20.30.40"
                                }
                            ],
                            "20": [
                                {
                                    "action": "permit",
                                    "log": "log",
                                    "src_host": "20.30.40.1"
                                }
                            ],
                            "30": [
                                {
                                    "action": "remark",
                                    "remark_name": "DEVICE - SNMP RW"
                                },
                                {
                                    "action": "permit",
                                    "src_ntw": "50.50.50.128",
                                    "src_wildcard": "0.0.0.127"
                                }
                            ],
                            "40": [
                                {
                                    "action": "permit",
                                    "src_ntw": "60.60.60.64",
                                    "src_wildcard": "0.0.0.63"
                                }
                            ]
                        }
                    ]
                }
            }
        }
    ]
]

I'm going to give a try to a file containing key/value pair for the complex regex

Need to enumerate all possible combinations or acl rules, pitfall - some of the regexes you put together might match same line, in that case TTP will select latest match, solution is to make your regexes as specific as possible together with results filtering using match variable contains and exclude functions for instance

This template:

<input load="text">
ip access-list standard 42
 10 remark machine_A
 10 permit 192.168.200.162
 20 remark machine_B
 20 permit 192.168.200.149
 30 deny   any log
ip access-list standard 98
 10 permit 10.10.10.1
 20 remark toto
 20 permit 30.30.30.1
 30 permit 30.30.30.0 0.0.0.255
ip access-list standard 99
 10 permit 10.20.30.40 log
 20 permit 20.30.40.1 log
 30 remark DEVICE - SNMP RW
 30 permit 50.50.50.128 0.0.0.127
 40 permit 60.60.60.64 0.0.0.63
ip access-list extended 199
 10 remark COLLECTOR - SNMP
 10 permit ip 70.70.70.0 0.0.0.255 any
 20 remark RETURN - Back
 20 permit ip 80.80.80.0 0.0.0.127 any
 30 remark VISUALIZE
 30 permit ip host 90.90.90.138 any
</input>

<group name="ip.{{ acl_type }}.{{ acl_name }}">
ip access-list {{ acl_type }} {{ acl_name }}
 <group name="{{ entry_id }}*" method="table">
 {{ entry_id }} remark {{ remark_name | re(".+") | let("action", "remark") }}
 {{ entry_id }} {{ action }} {{ src_host }}
 {{ entry_id }} {{ action }} {{ src_host | let("log", "log") }} log
 {{ entry_id }} {{ action }} {{ protocol }} host {{ src_host | let("dest_any", "any") }} any
 {{ entry_id }} {{ action }} {{ protocol }} {{ src_ntw | let("dest_any", "any") }} {{ src_wildcard | IP }} any
 {{ entry_id }} {{ action }} {{ src_ntw }} {{ src_wildcard | IP }}
 </group>
</group>

would produce this result:

[[{'ip': {'extended': {'199': {'10': [{'action': 'remark',
                                       'remark_name': 'COLLECTOR - SNMP'},
                                      {'action': 'permit',
                                       'dest_any': 'any',
                                       'protocol': 'ip',
                                       'src_ntw': '70.70.70.0',
                                       'src_wildcard': '0.0.0.255'}],
                               '20': [{'action': 'remark',
                                       'remark_name': 'RETURN - Back'},
                                      {'action': 'permit',
                                       'dest_any': 'any',
                                       'protocol': 'ip',
                                       'src_ntw': '80.80.80.0',
                                       'src_wildcard': '0.0.0.127'}],
                               '30': [{'action': 'remark',
                                       'remark_name': 'VISUALIZE'},
                                      {'action': 'permit',
                                       'dest_any': 'any',
                                       'protocol': 'ip',
                                       'src_host': '90.90.90.138'}]}},
          'standard': {'42': {'10': [{'action': 'remark',
                                      'src_host': 'machine_A'},
                                     {'action': 'permit',
                                      'src_host': '192.168.200.162'}],
                              '20': [{'action': 'remark',
                                      'remark_name': 'machine_B'},
                                     {'action': 'permit',
                                      'src_host': '192.168.200.149'}],
                              '30': [{'action': 'deny',
                                      'log': 'log',
                                      'src_host': 'any'}]},
                       '98': {'10': [{'action': 'permit',
                                      'src_host': '10.10.10.1'}],
                              '20': [{'action': 'remark',
                                      'remark_name': 'toto'},
                                     {'action': 'permit',
                                      'src_host': '30.30.30.1'}],
                              '30': [{'action': 'permit',
                                      'src_ntw': '30.30.30.0',
                                      'src_wildcard': '0.0.0.255'}]},
                       '99': {'10': [{'action': 'permit',
                                      'log': 'log',
                                      'src_host': '10.20.30.40'}],
                              '20': [{'action': 'permit',
                                      'log': 'log',
                                      'src_host': '20.30.40.1'}],
                              '30': [{'action': 'remark',
                                      'remark_name': 'DEVICE - SNMP RW'},
                                     {'action': 'permit',
                                      'src_ntw': '50.50.50.128',
                                      'src_wildcard': '0.0.0.127'}],
                              '40': [{'action': 'permit',
                                      'src_ntw': '60.60.60.64',
                                      'src_wildcard': '0.0.0.63'}]}}}}]]

Thanks with all those tips I'll update it in consequences.

If I'm able to reach a convenient structure I'll share it with you. It could be a good example to share with the community if you want to add it as example in the doc :-)

Let me know.

Anyway you can close this issue.

Regards,

Sam

Structure that might be not the most convenient one but worth checking is - RFC 8519 - YANG Data Model for Network Access Control Lists (ACLs).

Working now on adding yangsone module support in TTP outputters to validate produced results against YANG models. That might come in handy if you would decide to proceed with yang structures.

dmulyalin / ttp

Complex regex in vars are not working #36