search

dmulyalin / ttp

Template Text Parser
MIT License
350 stars 34 forks source link

ASA Template: Object-Groups Getting Mixed Up #55

Open consentfactory opened 3 years ago

consentfactory commented 3 years ago

Have a strange issue with an ASA template that I suspect is likely related to my template, but I'm unsure.

The gist is that I'm trying to get service and network object-groups into json, and generally they work, but I seem to have this issue where network group-objects end up in the service group objects.

Any tips on what I'm doing wrong (and how to improve it) would be great. Thanks in advance.

The code:

Input:
object-group service gokuhead
 service-object tcp-udp destination eq gokurpc 
 service-object tcp destination eq 902 
 service-object tcp destination eq https 
 service-object tcp destination eq nfs 
 service-object tcp destination eq 10025 
object-group network gohan
 network-object object gohan-01
 network-object object gohan-02
 network-object object vlan_944
 network-object object gohan-03
 network-object object gohan-05
 network-object object gohan-06
object-group service sql tcp
 port-object eq 1433
object-group network vegeta
 group-object trunks
 network-object object vegeta-01
object-group network Space-Users
 network-object object ab
 network-object object ac
 network-object object ad
 network-object object ae
 network-object object af
 network-object object ag
 network-object object ah
 network-object object ai
 network-object object aj
object-group network dalmatians
 network-object object dog-01
 group-object trunks
 network-object object vlan_950
 group-object Space-Users
 network-object object Darts-Summary

Template:
<vars>
SVC_PORTS = "tcp-udp|tcp|udp"
RANGE = "range"
OBJECT = "object"
</vars>

<group name="objects">

<group name="service-objects.{{svc_name}}**">
object service {{ svc_name | _line_ | joinmatches | _start_ }}
 description {{ description | ORPHRASE }}
 service {{protocol | re("SVC_PORTS") }} destination eq {{port}}
 service {{protocol | re("SVC_PORTS") }} destination range {{port_begin}} {{port_end}}
</group>

<group name="network-objects.{{net_name}}**">
object network {{ net_name | _line_ | joinmatches | _start_ }}
 description {{ description | ORPHRASE }}
 fqdn v4 {{fqdn}}
 host {{ip | IP}}
 host {{host | WORD}}
 subnet {{ip | IP}} {{ subnet | IP }}
 subnet {{ipv6 | PREFIXV6 }}
</group>

</group>

<group name="object-groups">

<group name="object-network-groups**.{{ network_name }}">
object-group network {{ network_name | _start_ }}
 description {{ description | ORPHRASE }}
 <group name="network-objects" itemize="obj_name">
 network-object object {{ obj_name }}
 network-object {{obj_name | exclude("OBJECT") }}
 </group>
 <group name="network-group-objects" itemize="net_obj_name">
 group-object {{ net_obj_name }}
 </group>
 <group name="network-object-networks*">
 network-object {{ ip | IP }} {{ subnet | IP }}
 </group>
 <group name="host-objects*" itemize="obj_name">
 network-object host {{ obj_name | WORD }}
 network-object host {{ obj_name | IP }}
 </group>
</group>

<group name="object-service-groups**.{{ service_name }}">

object-group service {{ service_name }} {{ protocol | re("SVC_PORTS") | _start_ }}
 description {{ description | ORPHRASE }}
 <group name="service-group-objects" itemize="svc_group_objs">
 group-object {{ svc_group_objs }}
 </group>
 <group name="service-port-objects" itemize="port_obj">
 port-object eq {{ port_obj }}
 </group>
 <group name="service-port-range*">
 port-object range {{ port_begin }} {{ port_end }}
 </group>
 <group name="service-protocol-objects" itemize="protocol_obj">
 protocol-object {{ protocol_obj }}
 </group>
object-group service {{ service_name | _start_ }}
 description {{ description | ORPHRASE }}
 <group name="service-group-objects" itemize="group_objs">
 group-object {{ group_objs }}
 </group>
 <group name="service-port-objects" itemize="port_obj">
 port-object eq {{ port_obj }}
 </group>
 <group name="service-port-range*">
 port-object range {{ port_begin }} {{ port_end }}
 </group>
 <group name="service-protocol-objects" itemize="protocol_obj">
 protocol-object {{ protocol_obj }}
 </group>
 <group name="service-objects*" itemize="object">
 service-object object {{ object }}
 service-object {{ object | exclude("OBJECT")}}
 </group>
 <group name="service-object-ports*">
 service-object {{ protocol | re("SVC_PORTS") }} destination eq {{port}}
 </group>
 <group name="service-object-port-ranges*">
 service-object {{ protocol | re("SVC_PORTS") }} destination range {{port_begin}} {{port_end}}
 </group>
</group>

<group name="object-protocol-groups**.{{ protocol_obj-group_name }}">
object-group protocol {{protocol_obj-group_name | _start_}}
 <group name="protocol-objects" itemize="protocol_object">
 protocol-object {{protocol_object}}
 </group>
</group>

</group>

Output:
[
    [
        {
            "object-groups": {
                "object-network-groups": {
                    "Space-Users": {
                        "network-objects": [
                            "ab",
                            "ac",
                            "ad",
                            "ae",
                            "af",
                            "ag",
                            "ah",
                            "ai",
                            "aj"
                        ]
                    },
                    "dalmatians": {
                        "network-objects": [
                            "dog-01",
                            "vlan_950",
                            "Darts-Summary"
                        ]
                    },
                    "gohan": {
                        "network-objects": [
                            "gohan-01",
                            "gohan-02",
                            "vlan_944",
                            "gohan-03",
                            "gohan-05",
                            "gohan-06"
                        ]
                    },
                    "vegeta": {
                        "network-group-objects": [
                            "trunks"
                        ],
                        "network-objects": [
                            "vegeta-01"
                        ]
                    }
                },
                "object-service-groups": {
                    "sql": {
                        "protocol": "tcp",
                        "service-group-objects": [
                            "trunks",
                            "Space-Users"
                        ],
                        "service-port-objects": [
                            "1433"
                        ]
                    }
                }
            }
        }
    ]
]
dmulyalin commented 3 years ago

Hi,

Try this template:

<vars>
SVC_PORTS = "tcp-udp|tcp|udp"
</vars>

<group name="object-{{ object_type }}-groups**.{{ object_name }}**">
object-group {{ object_type }} {{ object_name | _start_ }}
object-group {{ object_type }} {{ object_name | _start_ }} {{ protocol | re("SVC_PORTS")}}
 description {{ description | re(".*") }}

 <group name="network-objects" itemize="obj_name" method="table">
 network-object object   {{ obj_name | }}
 network-object host     {{ obj_name | IP }}
 </group> 

 <group name="group-objects" itemize="obj_name" method="table">
 group-object            {{ obj_name }}
 </group>

 <group name="group-objects" itemize="obj_name" method="table">
 service-object object   {{ obj_name }}
 service-object          {{ obj_name }}
 </group>

 <group name="service-object-ports*">
 service-object {{ protocol | re("SVC_PORTS") }} destination eq {{port}}
 </group>

 <group name="service-object-port-ranges*">
 service-object {{ protocol | re("SVC_PORTS") }} destination range {{port_begin}} {{port_end}}
 </group>

 <group name="service-port-objects" itemize="port_obj">
 port-object eq {{ port_obj }}
 </group>

</group>

it gives these results:

[[{'object-network-groups': {'Space-Users': {'network-objects': ['ab',
                                                                 'ac',
                                                                 'ad',
                                                                 'ae',
                                                                 'af',
                                                                 'ag',
                                                                 'ah',
                                                                 'ai',
                                                                 'aj']},
                             'dalmatians': {'group-objects': ['trunks',
                                                              'Space-Users'],
                                            'network-objects': ['dog-01',
                                                                'vlan_950',
                                                                'Darts-Summary']},
                             'gohan': {'network-objects': ['gohan-01',
                                                           'gohan-02',
                                                           'vlan_944',
                                                           'gohan-03',
                                                           'gohan-05',
                                                           'gohan-06']},
                             'vegeta': {'group-objects': ['trunks'],
                                        'network-objects': ['vegeta-01']}},
   'object-service-groups': {'gokuhead': {'service-object-ports': [{'port': 'gokurpc',
                                                                    'protocol': 'tcp-udp'},
                                                                   {'port': '902',
                                                                    'protocol': 'tcp'},
                                                                   {'port': 'https',
                                                                    'protocol': 'tcp'},
                                                                   {'port': 'nfs',
                                                                    'protocol': 'tcp'},
                                                                   {'port': '10025',
                                                                    'protocol': 'tcp'}]},
                             'sql': {'protocol': 'tcp',
                                     'service-port-objects': ['1433']}}}]]

Key difference is using dynamic path to encode object type as well:

<group name="object-{{ object_type }}-groups**.{{ object_name }}**">

In you original template lines

 group-object trunks
 group-object Space-Users

matched under object-service-groups because both object-service-groups and object-network-groups having this pattern defined:

 group-object {{ net_obj_name }}
 group-object {{ svc_group_objs }}

which translates to same regular expression.

TTP processes top object-service-groups and object-network-groups groups independently collecting all matches for them, combining these groups in a single group using dynamic path allows to obtain better results.

consentfactory commented 3 years ago

That makes sense, and I can actually see where I could use more dynamic pathing for some other templates.

Thank you for taking the time to create that and explain it!

consentfactory commented 3 years ago

Quick follow-up: is possible to combine matched items into a new object?

For example, objects in ASA configs can also appear as IPs and subnets:

object-group network Test-Group
 network-object 192.168.168.1 255.255.255.0

So I create a template item under the group like this:

network-object {{ obj_ip | IP }} {{ obj_subnet | to_cidr }}

But what would be really nice for my purposes is to concatenate obj_ip and obj_subnet into a new item called obj_name so they can appear in the list of network-objects.

Ideally, I would be able to add a backslash to have an IP in full CIDR notation like this: 192.168.168.1/24.

Is that possible? I haven't yet found documentation on something like this yet.

dmulyalin commented 3 years ago

Have a look at to_ip function

network-object {{ ip | PHRASE | to_ip | with_prefixlen }}

consentfactory commented 3 years ago

That's perfect. Thank you.

That said, in general is it possible to concatenate matches?

dmulyalin commented 3 years ago

There is joinmatches function available that can help in certain cases, also group function sformat can concatenate matches in a string, but in your case you not only need to concatenate but also translate mask to prefix length value, so only to_ip will do the trick.

consentfactory commented 3 years ago

I'll check that out. Thanks again.

consentfactory commented 3 years ago

I might have closed this early.

I can't seem to make this work.

network-object {{ ip | PHRASE | to_ip | with_prefixlen }}

Using this input:

object-group network gohan
 network-object 192.168.168.1 255.255.255.255
 network-object 192.168.168.2 255.255.255.255
 network-object host 192.168.168.3
 network-object object gohan-01
 network-object object gohan-02
 network-object object vlan_944
 network-object object gohan-03
 network-object object gohan-05
 network-object object gohan-06

I put the above template into this template:

<group name="object-{{ object_type }}-groups**.{{ object_name }}**">
object-group {{ object_type }} {{ object_name | _start_ }}
object-group {{ object_type }} {{ object_name | _start_ }} {{ protocol | re("SVC_PORTS")}}
 description {{ description | re(".*") }}

 <group name="network-objects" itemize="obj_name" method="table">
 network-object object {{ obj_name | WORD }}
 network-object host {{ obj_name | IP }}
 network-object {{ obj_name | PHRASE | to_ip | with_prefixlen }}
 </group>

 <group name="group-objects" itemize="obj_name" method="table">
 group-object            {{ obj_name }}
 </group>

 <group name="group-objects" itemize="obj_name" method="table">
 service-object object   {{ obj_name }}
 service-object          {{ obj_name }}
 </group>

 <group name="service-object-ports*">
 service-object {{ protocol | re("SVC_PORTS") }} destination eq {{port}}
 </group>

 <group name="service-object-port-ranges*">
 service-object {{ protocol | re("SVC_PORTS") }} destination range {{port_begin}} {{port_end}}
 </group>

 <group name="service-port-objects" itemize="port_obj">
 port-object eq {{ port_obj }}
 </group>

</group>

I get an error from the Python ipaddress module:

 ValueError: 'object/gohan-01' does not appear to be an IPv4 or IPv6 interface

# Also:

 ValueError: 'host/192.168.168.3' does not appear to be an IPv4 or IPv6 interface

What I don't understand is why network-object object {{ obj_name | WORD }} or network-object host {{ obj_name | IP }} isn't processing this. Those are more specific matches than the third template line network-object {{ obj_name | PHRASE | to_ip | with_prefixlen }}.

Once again, appreciate your help with this. Definitely learning a lot about ttp than what I knew before.

consentfactory commented 3 years ago

Was able to get the IP prefix to work by adding a variable with a regex pattern of words to exclude, then using that in the template to exclude. Is this the right approach?

<vars>
OBJECT_EXCLUDE = "host|object"
</vars>

<group name="object-groups">

<group name="object-{{ object_type }}-groups**.{{ object_name }}**">
object-group {{ object_type }} {{ object_name | _start_ }}
object-group {{ object_type }} {{ object_name | _start_ }} {{ protocol | re("SVC_PORTS")}}
 description {{ description | re(".*") }}

 <group name="network-objects" itemize="obj_name" method="table">
 network-object object {{ obj_name | WORD }}
 network-object host {{ obj_name | IP }}
 network-object {{ obj_name | PHRASE | exclude_re('OBJECT_EXCLUDE') | to_ip | with_prefixlen }}
 </group>
</group>
dmulyalin commented 3 years ago

Yes, that one of the options. You need to apply match filtering here, either using regexes or pattern check, for instance this should do the trick as well network-object {{ obj_name | PHRASE | contains(".") | to_ip | with_prefixlen }}

SudarshanVK commented 3 years ago

I echo what @consentfactory is saying.. i have learnt a lot from this thread.. @consentfactory : are you able to share the final template that you are using?

dmulyalin commented 3 years ago

@SudarshanVK would you mind sharing more info on these:

consentfactory commented 3 years ago

I echo what @consentfactory is saying.. i have learnt a lot from this thread.. @consentfactory : are you able to share the final template that you are using?

Yes, I have a template for the ASA, but I can't share it yet due to a project I'm working on. I'll update this thread with it when I upload the template*.

SudarshanVK commented 3 years ago

@dmulyalin : i am not able to share the configuration file unfortunately. But, here is the template i am using so far. I am yet to work out a template for ACL, user groups, OSPF configuration. I am hoping for a structure that would be easy to output to a spreadsheet for further analysis.

PS: I started looking into TTP only 4 days ago.

<vars>
SVC_PORTS = "tcp-udp|tcp|udp"
OBJECT_EXCLUDE = "host|object"
</vars>

<group name="global">
hostname {{hostname}}
</group>

<group name="interface.PortChannel">
interface Port-channel{{ number | DIGIT }}.{{ subinterface | DIGIT }}
 nameif {{ name }}
 security-level {{ security_level }}
 ip address {{ ip | PHRASE | to_ip | with_prefixlen }}
 ospf network {{ ospf_type }}
</group>

<group name="object-{{ object_type }}-groups**.{{ object_name }}**">
object-group {{ object_type }} {{ object_name | _start_ }}
object-group {{ object_type }} {{ object_name | _start_ }} {{ protocol | re("SVC_PORTS")}}
 description {{ description | re(".*") }}

 <group name="network-objects" itemize="obj_name" method="table">
 network-object object   {{ obj_name | WORD}}
 network-object host     {{ obj_name | IP }}
 network-object {{ obj_name | PHRASE | exclude_re('OBJECT_EXCLUDE') | to_ip | with_prefixlen }}
 </group> 

 <group name="group-objects" itemize="obj_name" method="table">
 group-object            {{ obj_name }}
 </group>

 <group name="group-objects" itemize="obj_name" method="table">
 service-object object   {{ obj_name }}
 service-object          {{ obj_name }}
 </group>

 <group name="service-object-ports*">
 service-object {{ protocol | re("SVC_PORTS") }} destination eq {{port}}
 </group>

 <group name="service-object-port-ranges*">
 service-object {{ protocol | re("SVC_PORTS") }} destination range {{port_begin}} {{port_end}}
 </group>

 <group name="service-port-objects" itemize="port_obj">
 port-object eq {{ port_obj }}
 </group>

</group>

<output
format="excel"
returner="file"
filename="ASA_report.xslx"
url="./"
load="yaml"
>
table:
  - path: interface.PortChannel
    tab_name: PortChannel
</output>
dmulyalin commented 3 years ago

@SudarshanVK That template of yours is quiet sophisticated, looks like you picked up quiet a bit of TTP so far.

What is the problem with above template, what exactly does not work in it for you? Also, would strongly advise against sharing any actual devices configs here, good idea is to anonymize them before doing so.

Will need sample data with template and desired results structure to help with building the template, without it cannot do much apart from general advice.

If you Guys finish doing the template for ASA, mind to think about contributing them to TTP Templates repo, if you think that it would be good for other people to benefit from your work.

structure that would be easy to output to a spreadsheet - that structure should be a list of dictionaries in that case, make sure to align your templates toward that format.

SudarshanVK commented 3 years ago

@dmulyalin i have a 80 context firewall to migrate.. I am still developing the template and absolutely i am keen to share it with everyone once i have everything working.. I might reach-out over here in case i run into issues with the rest of the configuration that I am trying to Parse... Thank you thus far...

consentfactory commented 3 years ago

If you Guys finish doing the template for ASA, mind to think about contributing them to TTP Templates repo, if you think that it would be good for other people to benefit from your work.

I would be more than happy to help contribute to the templates.

That said, the most difficult problem I face with ASA configs is the access-list configurations. I'm not sure if there are issues with the way my template is written (possible), or if the configs are just that complicated because there are lots, and I mean lots, of variations in how the configuration can be written, and I'm not sure I've captured all of them ( I think I have ~100 lines of variations, not including another set of lines that append these lines with 'log debugging').

Would be great to open-source that for someone else to take a look and improve them.

consentfactory commented 3 years ago

Here is the ASA template I've put together: https://gist.github.com/consentfactory/85872fc83453d1735b15aed3e47a9763

showipintbri commented 3 years ago

@consentfactory I'm having some errors trying to load your template. This is even before I pass it any configs to parse.

from ttp import ttp

ttp_template = """
[insert contents of your linked gist from above]
"""

parser = ttp()
parser.add_template(ttp_template)

The resulting error:

Traceback (most recent call last):
  File "C:\Users\my_username\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\ttp\ttp.py", line 1324, in construct_etree
    template_ET = ET.XML(template_text)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\xml\etree\ElementTree.py", line 1320, in XML
    parser.feed(text)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 2, column 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:/Users/my_username/Desktop/Python/New Stuff/ttp_test.py", line 368, in <module>
    parser.add_template(ttp_template)
  File "C:\Users\my_username\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\ttp\ttp.py", line 347, in add_template
    template_obj = _template_class(
  File "C:\Users\my_username\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\ttp\ttp.py", line 906, in __init__
    self.template = self.handle_extend(template_text)
  File "C:\Users\my_username\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\ttp\ttp.py", line 1278, in handle_extend
    template_ET = self.construct_etree(template_text)
  File "C:\Users\my_username\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\ttp\ttp.py", line 1331, in construct_etree
    template_ET = ET.XML("<template>\n{}\n</template>".format(template_text))    
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.2800.0_x64__qbz5n2kfra8p0\lib\xml\etree\ElementTree.py", line 1320, in XML
    parser.feed(text)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 26, column 9

Any ideas?

showipintbri commented 3 years ago

^Ignore.

I closed my laptop up for the day and went home.

Once home, I re-did everything and it is all now working. 🤷 Operator error? possibly.