dmulyalin / ttp

Template Text Parser
MIT License
350 stars 34 forks source link

match within a group is repeated in subsequent group #37

Closed chrispella closed 3 years ago

chrispella commented 3 years ago

We are trying to parse the configuration from a Nokia 7750 SR router. So far we have had good luck with ttp and are looking to use it to replace some messy and fragile collections of regexes. However we have run into a issue where we have a matched pattern that is being repeated in the next group and haven't been able to find out whether it is our template or a bug. For instance, if you look at the output the sdp with id 8035 from epipe id 103206 shows up in the next epipe group 103256. The formatting is off below unless you open it to edit ... Is there a way to preserve the spacing?

Here is our template:

<macro>
import re
def qinq(data):
    data = re.sub(r"\*", r"qinq", data)
    return data 
</macro>

<group name="service">
    service {{ ignore }}
    <group name="epipe.{{ service_id }}"  default="none">
        epipe {{ service_id | _start_ }} customer {{ customer_id }} create
            description "{{ description | ORPHRASE | default("none") }}"
            service-mtu {{ service_mtu | default("none") }}
            service-name "{{ service_name | ORPHRASE | default("none") }}"
        <group name="endpoint"  default="none">
            endpoint {{ endpoint | _start_ }} create
                revert-time {{ revert_time | default("none") }}
            exit {{ _end_ }}
        </group>     
        <group name="sap.{{ sap_id }}"  default="none">
            sap {{ sap_id | macro("qinq") | _start_ | ORPHRASE }} create
                description "{{ description | ORPHRASE | default("none")}}"
                multi-service-site "{{ mss_name | default("none") }}"
            <group name="ingress" default="default_ingress" >
                ingress {{ _start_ }}
                    qos {{ sap_ingress | default("1") }}                 
                    scheduler-policy {{ scheduler_policy | default("none")}}
                exit {{ _end_ }}
            </group>
            <group name="egress" default="default_egress">
                egress {{ _start_ }}
                    scheduler-policy {{ scheduler_policy | default("none") }}
                    qos {{ sap_egress | default("1)") }}
                exit {{ _end_ }}
            </group>
                no shutdown {{ state | set("enabled") | default("disabled") }}
            exit {{ _end_ }}
        </group>
        <group name="pwr_sdp.{{pwr_spoke_sdp_id}}**" default="none"> 
            spoke-sdp {{ pwr_spoke_sdp_id | default("none")}}:{{vc_id | _start_ | default("none") }} endpoint {{ endpoint | default("none") }} create             
                precedence {{ precedence | default("default_precedence") }}
                no shutdown {{ state | set("enabled") | default("disabled") }}
            exit {{ _end_ }}
        </group> 
        <group name="regular_sdp.{{r_spoke_sdp_id}}**" default="none"> 
            spoke-sdp {{ r_spoke_sdp_id }}:{{vc_id | _start_ }} create             
                no shutdown {{ state | set("enabled") | default("disabled") }}
            exit {{ _end_ }}
        </group> 
            no shutdown {{ state | set("enabled") | default("disabled") }}
        exit {{ _end_ }}
    </group>
    exit {{ _end_ }}
</group>

Here is an extract of the config file which we are using:

 epipe 103076 customer 160 create
            description "vf=EWL:cn=TATA_COM:tl=2C02495918:st=act:"
            service-mtu 1588
            service-name "EPIPE service-103076 DKTN08a-D0105 (63.130.108.41)"
            sap 1/2/12:20.* create
                description "vf=EWL:cn=TATA_COM:tl=2C02495890:st=act:"
                multi-service-site "TATA_VSNL_STRAT_A206_LAN10"
                ingress
                    queue-override
                        queue 1 create
                            cbs default
                            mbs 40 kilobytes
                            rate 10000 cir 10000
                        exit
                    exit
                exit
                egress
                    queue-override
                        queue 1 create
                            cbs default
                            mbs 40 kilobytes
                            rate 10000 cir 10000
                        exit
                    exit
                exit
                accounting-policy 4
                no shutdown
            exit
            spoke-sdp 8051:103076 create
                no shutdown
            exit
            no shutdown
        exit
        epipe 103206 customer 1904 create
            description "vf=1273:cn=skanska:tl=3C02407455:st=act:no='SKANSKA UK PLC Stepney Green E1 3DG'"
            service-mtu 1988
            service-name "EPIPE service-103206 DKTN08a-D0105 (63.130.108.41)"
            sap 2/2/3:401.100 create
                description "vf=1273:cn=skanska:tl=3C02407455:st=act:no='SKANSKA UK PLC Stepney Green E1 3DG'"
                multi-service-site "SKANSKA_E13DG_A825_LAN1"
                ingress
                    qos 11010
                    queue-override
                        queue 1 create
                            cbs default
                            mbs 1188 kilobytes
                            rate max cir 47500
                        exit
                        queue 3 create
                            cbs default
                            mbs 63 kilobytes
                            rate max cir 2500
                        exit
                    exit
                exit
                egress
                    qos 11010
                    queue-override
                        queue 1 create
                            cbs default
                            mbs 1188 kilobytes
                            rate max cir 47500
                        exit
                        queue 3 create
                            cbs default
                            mbs 63 kilobytes
                            rate max cir 2500
                        exit
                    exit
                exit
                collect-stats
                accounting-policy 4
                no shutdown
            exit
            spoke-sdp 8035:103206 create
                no shutdown
            exit
            no shutdown
        exit
        epipe 103256 customer 160 create
            description "vf=EWL:cn=TATA_COMM:tl=2C02490189:st=act:"
            service-mtu 1988
            service-name "EPIPE service-103256 DKTN08a-D0105 (63.130.108.41)"
            sap 1/2/12:15.* create
                description "vf=EWL:cn=TATA_COMM:tl=2C02490171:st=act:"
                multi-service-site "TATA_VSNL_STRAT_A206_LAN5"
                ingress
                    qos 11000
                    queue-override
                        queue 1 create
                            cbs default
                            mbs 391 kilobytes
                            rate 100000 cir 100000
                        exit
                    exit
                exit
                egress
                    qos 11000
                    queue-override
                        queue 1 create
                            cbs default
                            mbs 391 kilobytes
                            rate 100000 cir 100000
                        exit
                    exit
                exit
                accounting-policy 4
                no shutdown
            exit
            spoke-sdp 8139:103256 create
                no shutdown
            exit
            no shutdown
        exit
        epipe 103742 customer 160 create
            description "vf=EWL:cn=TATA_COM:tl=2C02410363:st=act:"
            service-mtu 1588
            service-name "EPIPE service-103742 DKTN08a-D0105 (63.130.108.41)"
            sap 5/2/50:20.* create
                description "vf=EWL:cn=TATA_COM:tl=2C02410338:st=act:"
                multi-service-site "TATA_STRAT_LON_A206_LANA"
                ingress
                    qos 11000
                    queue-override
                        queue 1 create
                            cbs default
                            mbs 32 kilobytes
                            rate 8000 cir 8000
                        exit
                    exit
                exit
                egress
                    qos 11000
                    queue-override
                        queue 1 create
                            cbs default
                            mbs 32 kilobytes
                            rate 8000 cir 8000
                        exit
                    exit
                exit
                accounting-policy 4
                no shutdown
            exit
            spoke-sdp 8061:103742 create
                no shutdown
            exit
            no shutdown
        exit
        epipe 55513386 customer 4 vc-switching create
            description "vf=EAGG:cn=Bulldog:tl=VF"
            service-mtu 1526
            spoke-sdp 78:55513386 create
                control-word
                no shutdown
            exit
            spoke-sdp 8245:55513386 create
                control-word
                no shutdown
            exit
            no shutdown
        exit
        epipe 55517673 customer 4 create
            description "vf=EAGG:cn=Bulldog:tl=2C01291821:st=act:no=NGA EPIPE#BAACTQ#VLAN 901"
            service-mtu 1526
            service-name "epipe service-64585 DKTN08a-D0105 (63.130.108.41)"
            endpoint "SDP" create
                revert-time infinite
            exit
            sap 2/2/3:901.* create
                description "2_2_3,H0505824A,Bulldog,VLAN 901"
                ingress
                    scheduler-policy "NGA-LLU-300M"
                    qos 20010
                exit
                egress
                    scheduler-policy "NGA-LLU-300M"
                    qos 20010
                exit
                no shutdown
            exit
            spoke-sdp 8243:55517673 endpoint "SDP" create
                collect-stats
                precedence 1
                no shutdown
            exit
            spoke-sdp 8245:55517673 endpoint "SDP" create
                collect-stats
                precedence primary
                no shutdown
            exit
            no shutdown
        exit     

What we see when we parse it is that the one

parser.py:16 <module>
    parser.result()[0][0]: {
        'service': {
            'epipe': {
                '103076': {
                    'service_name': 'EPIPE service-103076 DKTN08a-D0105 (63.130.108.41)',
                    'service_mtu': '1588',
                    'description': 'vf=EWL:cn=TATA_COM:tl=2C02495918:st=act:',
                    'customer_id': '160',
                    'state': 'disabled',
                    'sap': {
                        '1/2/12:20.qinq': {
                            'mss_name': 'TATA_VSNL_STRAT_A206_LAN10',
                            'description': 'vf=EWL:cn=TATA_COM:tl=2C02495890:st=act:',
                            'state': 'enabled',
                            'ingress': {
                                'sap_ingress': '1',
                                'scheduler_policy': 'none',
                            },
                            'egress': {
                                'scheduler_policy': 'none',
                                'sap_egress': '1)',
                            },
                        },
                    },
                    'regular_sdp': {
                        '8051': {
                            'state': 'enabled',
                            'vc_id': '103076',
                        },
                    },
                },
                '103206': {
                    'service_name': 'EPIPE service-103206 DKTN08a-D0105 (63.130.108.41)',
                    'service_mtu': '1988',
                    'description': "vf=1273:cn=skanska:tl=3C02407455:st=act:no='SKANSKA UK PLC Stepney Green E1 3DG'",
                    'customer_id': '1904',
                    'state': 'enabled',
                    'sap': {
                        '2/2/3:401.100': {
                            'mss_name': 'SKANSKA_E13DG_A825_LAN1',
                            'description': "vf=1273:cn=skanska:tl=3C02407455:st=act:no='SKANSKA UK PLC Stepney Green E1 3DG'",
                            'state': 'disabled',
                            'ingress': {
                                'sap_ingress': '11010',
                                'scheduler_policy': 'none',
                            },
                            'egress': {
                                'sap_egress': '11010',
                                'scheduler_policy': 'none',
                            },
                        },
                    },
                    **'regular_sdp': {
                        '8035': {
                            'state': 'enabled',
                            'vc_id': '103206',
                        },**
                    },
                },
                '103256': {
                    'service_name': 'EPIPE service-103256 DKTN08a-D0105 (63.130.108.41)',
                    'service_mtu': '1988',
                    'description': 'vf=EWL:cn=TATA_COMM:tl=2C02490189:st=act:',
                    'customer_id': '160',
                    'state': 'enabled',
                    'sap': {
                        '1/2/12:15.qinq': {
                            'mss_name': 'TATA_VSNL_STRAT_A206_LAN5',
                            'description': 'vf=EWL:cn=TATA_COMM:tl=2C02490171:st=act:',
                            'state': 'disabled',
                            'ingress': {
                                'sap_ingress': '11000',
                                'scheduler_policy': 'none',
                            },
                            'egress': {
                                'sap_egress': '11000',
                                'scheduler_policy': 'none',
                            },
                        },
                    },
                    'regular_sdp': {
                        '8035': {
                            'state': 'enabled',
                        },
                        '8139': {
                            'state': 'enabled',
                            'vc_id': '103256',
                        },
                    },
                },
            },
        },
    } (dict) len=1
dmulyalin commented 3 years ago

Thanks for raising that issue, it was a bug in internal logic, fixed in latest commit b6694f840324897e64d70f24719cc7db511cd2fb, feel free to clone and install to test it.

In return, cleaned up your template a bit, as these things either not required or do nothing:

<group name="service">
    service {{ ignore }}
    <group name="epipe.{{ service_id }}"  default="none">
        epipe {{ service_id }} customer {{ customer_id }} create
            description "{{ description | ORPHRASE }}"
            service-mtu {{ service_mtu }}
            service-name "{{ service_name | ORPHRASE }}"
        <group name="endpoint"  default="none">
            endpoint {{ endpoint }} create
                revert-time {{ revert_time }}
            exit {{ _end_ }}
        </group>     
        <group name="sap.{{ sap_id }}"  default="none">
            sap {{ sap_id | resub(r"\*", "qinq") | ORPHRASE }} create
                description "{{ description | ORPHRASE }}"
                multi-service-site "{{ mss_name }}"
            <group name="ingress">
                ingress {{ _start_ }}
                    qos {{ sap_ingress | default("1") }}                 
                    scheduler-policy {{ scheduler_policy | default("none")}}
                exit {{ _end_ }}
            </group>
            <group name="egress">
                egress {{ _start_ }}
                    scheduler-policy {{ scheduler_policy | default("none") }}
                    qos {{ sap_egress | default("1)") }}
                exit {{ _end_ }}
            </group>
                no shutdown {{ state | set("enabled") | default("disabled") }}
            exit {{ _end_ }}
        </group>
        <group name="pwr_sdp.{{pwr_spoke_sdp_id}}**" default="none"> 
            spoke-sdp {{ pwr_spoke_sdp_id }}:{{vc_id }} endpoint {{ endpoint }} create             
                precedence {{ precedence | default("default_precedence") }}
                no shutdown {{ state | set("enabled") | default("disabled") }}
            exit {{ _end_ }}
        </group> 
        <group name="regular_sdp.{{r_spoke_sdp_id}}**" default="none"> 
            spoke-sdp {{ r_spoke_sdp_id }}:{{vc_id }} create             
                no shutdown {{ state | set("enabled") | default("disabled") }}
            exit {{ _end_ }}
        </group> 
            no shutdown {{ state | set("enabled") | default("disabled") }}
        exit {{ _end_ }}
    </group>
    exit {{ _end_ }}
</group>
chrispella commented 3 years ago

Thanks so much. It works fine on our end. As you noticed we are still learning what is required with the templates

chrispella commented 3 years ago

Working now.

chrispella commented 3 years ago

Oh, maybe I shouldn't close it until it is merged into master?

dmulyalin commented 3 years ago

Fix already merged in master brunch and will be included in next release.

Thanks again for raising that problem, let me know if you'll have any further issues, will see what I can do.