dmulyalin / ttp

Template Text Parser
MIT License
351 stars 34 forks source link

v0.8.0 unexpected parsing results #57

Open iskobkarev opened 3 years ago

iskobkarev commented 3 years ago

Hello I'm new to ttp and I find it extremely useful for managing network equipment.

I've hit a few bumps and I wonder if that is an issue or I'm missing something

I'm using http://textfsm.nornir.tech/ to develop/troubleshoot templates and the templates work flawlessly there. When used on my ubuntu (Linux mgmtuntu 5.4.0-80-generic) I'm getting lots of issues with parsing. root@mgmtuntu:/home/admin1/ttp/portint# python3 --version Python 3.8.10

A simple example with HPE Compare switch output (disp int brief, truncated to include just a few interfaces ):

device output: Brief information on interfaces in route mode: Link: ADM - administratively down; Stby - standby Protocol: (s) - spoofing Interface Link Protocol Primary IP Description
InLoop0 UP UP(s) --
REG0 UP -- --
Vlan401 UP UP 10.251.147.36 HSSBC_to_inband_mgmt_r4

Brief information on interfaces in bridge mode: Link: ADM - administratively down; Stby - standby Speed: (a) - auto Duplex: (a)/A - auto; H - half; F - full Type: A - access; T - trunk; H - hybrid Interface Link Speed Duplex Type PVID Description
BAGG1 UP 20G(a) F(a) T 1 to-KDC-R4.10-Core-1 BAGG14 UP 10G(a) F(a) T 1 KDC-R429-E1 BackUp Chassis BAGG22 UP 20G(a) F(a) T 1 HSSBC-NS-01 FGE1/0/49 DOWN auto A A 1
XGE1/0/1 UP 10G(a) F(a) T 1 KDC-R402-E1 Backup Chassis

Template:

Brief information on interfaces in route mode: {{ _start_ }} Interface Link Protocol Primary_IP Description {{ _headers_ }} {{ _end_}} Brief information on interfaces in bridge mode: {{ _start_}} Interface Link Speed Duplex Type PVID Description {{ _headers_}} {{ _end_}}

Result with the nornir.tech ( works as expected) [ { "interfaces": { "bridged": { "BAGG1": { "Description": "to-KDC-R4.10-Core-1", "Duplex": "F(a)", "Link": "UP", "PVID": "1", "Speed": "20G(a)", "Type": "T" }, "BAGG14": { "Description": "KDC-R429-E1 BackUp Chassis", "Duplex": "F(a)", "Link": "UP", "PVID": "1", "Speed": "10G(a)", "Type": "T" }, "BAGG22": { "Description": "HSSBC-NS-01", "Duplex": "F(a)", "Link": "UP", "PVID": "1", "Speed": "20G(a)", "Type": "T" }, "FGE1/0/49": { "Description": "", "Duplex": "A", "Link": "DOWN", "PVID": "1", "Speed": "auto", "Type": "A" }, "XGE1/0/1": { "Description": "KDC-R402-E1 Backup Chassis", "Duplex": "F(a)", "Link": "UP", "PVID": "1", "Speed": "10G(a)", "Type": "T" } }, "routed": { "InLoop0": { "Description": "", "Link": "UP", "Primary_IP": "--", "Protocol": "UP(s)" }, "REG0": { "Description": "", "Link": "UP", "Primary_IP": "--", "Protocol": "--" }, "Vlan401": { "Description": "HSSBC_to_inband_mgmt_r4", "Link": "UP", "Primary_IP": "10.251.147.36", "Protocol": "UP" } } } } ]

Cli tool / ttp 0.5-0.6-0.7-0.8 produces mixed up output

root@mgmtuntu:/home/admin1/ttp/portint# ttp --data "cmw7-disp-int-brief.txt" --template "../templates/hp_cmw7_display_interface_brief.ttp" -l ERROR -o yaml

iskobkarev commented 3 years ago

some tags were removed when the template was posted. Attaching as a file. hp_cmw7_display_interface_brief.txt

iskobkarev commented 3 years ago

more testing done. 0.4.0 is the last one to parse this template "correctly". 0.5 and above seem to not do it

0.4 debug :

                        'REGEX': re.compile('\\nBrief\\ +information\\ +on\\ +interfaces\\ +in\\ +route\\ +mode:[\\t ]*(?=\\n)'),

0.8 debug:

                        'REGEX': re.compile('\\nBrief[ \\t]+information[ \\t]+on[ \\t]+interfaces[ \\t]+in[ \\t]+route[ \\t]+mode:[\\t ]*(?=\\n|\\r\\n)'),
dmulyalin commented 3 years ago

@iskobkarev thank you for raising this issue, could you share sample data in a form of text file similar to how you did with ttp template, sample data you provided above did not preserve spacing between words.

iskobkarev commented 3 years ago

Please see attached src_short.txt

dmulyalin commented 3 years ago

Introduced fix in internal logic in commit a228209350cd6dcc12e629fe8d5addaa290ae04f, this should not scramble results that much as it was before, however, still need to think about how to handle headers indicator, as in current state it will match some undesired results which can be removed using group macro with some filtering. This is the results produced after this fix: https://github.com/dmulyalin/ttp/blob/a228209350cd6dcc12e629fe8d5addaa290ae04f/test/pytest/test_answers_and_docs.py#L5144

The reason why it was working in TTP 0.4.0 is due to the diffeence in how heders indicator regex formed:

0.4.0
'REGEX': re.compile('\\n(?P<Interface>.{21})(?P<Link>.{5})(?P<Protocol>.{9})(?P<Primary_IP>.{16})(?P<Description>.{11,})(?=\\n)')
'REGEX': re.compile('\\n(?P<Interface>.{21})(?P<Link>.{5})(?P<Speed>.{8})(?P<Duplex>.{7})(?P<Type>.{5})(?P<PVID>.{5})(?P<Description>.{11,})(?=\\n)') 

0.8.0
'REGEX': re.compile('\\n(?P<Interface>.{21})(?P<Link>.{5})(?P<Protocol>.{9})(?P<Primary_IP>.{1,16})(?P<Description>.*)(?=\\n|\\r\\n)'),
'REGEX': re.compile('\\n(?P<Interface>.{21})(?P<Link>.{5})(?P<Speed>.{8})(?P<Duplex>.{7})(?P<Type>.{5})(?P<PVID>.{1,5})(?P<Description>.*)(?=\\n|\\r\\n)'),

probably will add some variable for headers, something like {{ _headers_ | strict }} to introduce same behaviour as in TTP 0.4.0.

In a meantime feel free to install TTP from master - python3 -m pip install git+https://github.com/dmulyalin/ttp - and give it a go

But try this template:

<group name = "interfaces">
<group name="routed">
Brief information on interfaces in route mode: {{ _start_ }}
<group name = "{{Interface}}">
Interface            Link Protocol Primary_IP      Description {{ _headers_ }}
</group>
{{ _end_ }}
</group>

<group name="bridged">
Brief information on interfaces in bridge mode: {{ _start_ }}
<group name = "{{Interface}}">
Interface            Link Speed   Duplex Type PVID Description {{ _headers_ }}
</group>
{{ _end_ }}
</group>
</group>

As it has different position for {{ _end_ }} as compared to your original template essentially indicating that <group name="routed"> or <group name="bridged"> ended

iskobkarev commented 3 years ago

Hello

Thank you for your prompt response. It looks better now and it properly matches the routed vs bridged with the spaces in the start line. This particular issue seem to be resolved.

I have one more question though. There are two other lines after the start line and ttp treats one of these lines "Link: ADM - administratively down; Stby - standby" as the part of the interface table group producing an invalid entry: routed: InLoop0: Description: '' Link: UP Primary_IP: -- Protocol: UP(s) 'Link: ADM - administr': !!-> treats this like an interface Description: '' Link: ative Primary_IP: Stby - standby Protocol: ly down; REG0:

This is the input text:

Brief information on interfaces in route mode: !! start matching line Link: ADM - administratively down; Stby - standby !! -> the line which needs to be ignored Protocol: (s) - spoofing Interface Link Protocol Primary IP Description InLoop0 UP UP(s) --

For this particular output I have a workaround (see below) to match the line immediately preceding the first interface entry but that might not be possible in some cases and it would be useful to be able to discard some lines based on the whole line content or a regex match. {{ ignore("line") }} does not seem to catch it. Am I using it properly or am I missing something ? Here is a snippet from the final script which works fine for me matching the line immediately preceding the interface table Interface Link Protocol Primary IP Description {{ _start_ }} Interface Link Protocol Primary_IP Description {{ _headers_ }} {{ _end_ }} Thanks again Igor On Thu, Sep 16, 2021 at 7:18 AM dmulyalin ***@***.***> wrote: > Introduced fix in internal logic in commit a228209 > , > this should not scramble results that much as it was before, however, still > need to think about how to handle headers indicator, as in current state it > will match some undesired results which can be removed using group macro > with some filtering. This is the results produced after this fix: > https://github.com/dmulyalin/ttp/blob/a228209350cd6dcc12e629fe8d5addaa290ae04f/test/pytest/test_answers_and_docs.py#L5144 > > The reason why it was working in TTP 0.4.0 is due to the diffeence in how > heders indicator regex formed: > > 0.4.0 > 'REGEX': re.compile('\\n(?P.{21})(?P.{5})(?P.{9})(?P.{16})(?P.{11,})(?=\\n)') > 'REGEX': re.compile('\\n(?P.{21})(?P.{5})(?P.{8})(?P.{7})(?P.{5})(?P.{5})(?P.{11,})(?=\\n)') > > 0.8.0 > 'REGEX': re.compile('\\n(?P.{21})(?P.{5})(?P.{9})(?P.{1,16})(?P.*)(?=\\n|\\r\\n)'), > 'REGEX': re.compile('\\n(?P.{21})(?P.{5})(?P.{8})(?P.{7})(?P.{5})(?P.{1,5})(?P.*)(?=\\n|\\r\\n)'), > > probably will add some variable for headers, something like {{ _headers_ > | strict }} to introduce same behaviour as in TTP 0.4.0. > > In a meantime feel free to install TTP from master - python3 -m pip > install git+https://github.com/dmulyalin/ttp - and give it a go > > But try this template: > > > > Brief information on interfaces in route mode: {{ _start_ }} > > Interface Link Protocol Primary_IP Description {{ _headers_ }} > > {{ _end_ }} > > > > Brief information on interfaces in bridge mode: {{ _start_ }} > > Interface Link Speed Duplex Type PVID Description {{ _headers_ }} > > {{ _end_ }} > > > > As it has different position for {{ _end_ }} as compared to your original > template essentially indicating that or name="bridged"> ended > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > , or > unsubscribe > > . > Triage notifications on the go with GitHub Mobile for iOS > > or Android > . > > -- Thanks, Igor Skobkarev, ***@***.***
dmulyalin commented 3 years ago

You need to do post processing filtering of matched results using group macro referring to custom python function, have a look at this comment for the idea - https://github.com/dmulyalin/ttp/issues/15#issuecomment-665281850

However, now another option available, added headers new feature - columns variable, that allows to turn on behaviour similar to TTP 0.4.0 headers implementation, commit - b6d19536a1c84705f253a9f4a03b654d30671270, have a look at this testcase for sample usage: https://github.com/dmulyalin/ttp/blob/b6d19536a1c84705f253a9f4a03b654d30671270/test/pytest/test_answers_and_docs.py#L5367

iskobkarev commented 3 years ago

Hi Denis

Thanks again for the TTP, it is a great tool and I am doing quite a bit with it already.

I hit a bit of a roadblock and I'm not sure what I'm missing here. The files with full scripts and outputs are attached.

in short, this group without "*" after the group name produces the desired results except when there is just one member it does not create a list (which is probably not a big deal)

Local: {{_start_}} {{ ignore(" Port.*") }} {{ ignore("-+") }} {{interface}} {{status}} {{priority}} {{oper_key}} {{flag}}

When I add a "*" after the group name it always produces a list but it contains some empty elements - likely due to ignores.

Local: {{_start_}} {{ ignore(" Port.*") }} {{ ignore("-+") }} {{interface}} {{status}} {{priority}} {{oper_key}} {{flag}}

I wonder if there is a way to always produce a list but without empty elements.

Thanks

Igor

On Sat, Sep 18, 2021 at 4:05 AM dmulyalin @.***> wrote:

You need to do post processing filtering of matched results using group macro referring to custom python function, have a look at this comment for the idea - #15 (comment) https://github.com/dmulyalin/ttp/issues/15#issuecomment-665281850

However, now another option available, added headers new feature - columns variable, that allows to turn on behaviour similar to TTP 0.4.0 headers implementation, commit - b6d1953 https://github.com/dmulyalin/ttp/commit/b6d19536a1c84705f253a9f4a03b654d30671270, have a look at this testcase for sample usage: https://github.com/dmulyalin/ttp/blob/b6d19536a1c84705f253a9f4a03b654d30671270/test/pytest/test_answers_and_docs.py#L5367

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmulyalin/ttp/issues/57#issuecomment-922259955, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATJNEU3QSGH6MRTZ5ONGISDUCRXARANCNFSM5EDGLZDA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Thanks,

Igor Skobkarev, @.***

@.***:/home/admin1/ttp/smhcore# more lagttp.ttp

Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing Port Status: S -- Selected, U -- Unselected, I -- Individual, * -- Management port Flags: A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation, D -- Synchronization, E -- Collecting, F -- Distributing, G -- Defaulted, H -- Expired

Aggregate Interface: Bridge-Aggregation1 Aggregation Mode: Dynamic Loadsharing Type: Shar Management VLAN : None System ID: 0x8000, d07e-28b5-a200 Local: Port Status Priority Oper-Key Flag

GE6/0/1 U 32768 1 {ACG} GE6/0/2 U 32768 1 {ACG} Remote: Actor Partner Priority Oper-Key SystemID Flag

GE6/0/1 0 32768 0 0x8000, 0000-0000-0000 {EF} GE6/0/2 0 32768 0 0x8000, 0000-0000-0000 {EF}

Aggregate Interface: Bridge-Aggregation2 Aggregation Mode: Dynamic Loadsharing Type: Shar Management VLAN : None System ID: 0x8000, d07e-28b5-a200 Local: Port Status Priority Oper-Key Flag

GE6/0/3 U 32768 2 {ACG} GE6/0/4 U 32768 2 {ACG} Remote: Actor Partner Priority Oper-Key SystemID Flag

GE6/0/3 0 32768 0 0x8000, 0000-0000-0000 {EF} GE6/0/4 0 32768 0 0x8000, 0000-0000-0000 {EF}

Aggregate Interface: Bridge-Aggregation{{ channel_number}} Local: {{_start_}} {{ ignore(" Port.*") }} {{ ignore("-+") }} {{interface}} {{status}} {{priority}} {{oper_key}} {{flag}} Remote: {{ _start_ }} {{ ignore(" Port.*")}} {{ ignore("-+")}} {{interface}} {{status}} {{priority}} {{oper_key}} {{sys_id}}, {{ mac }} {{flag}}

@.***:/home/admin1/ttp/smhcore# ttp -t lagttp.ttp -o yaml

@.***:/home/admin1/ttp/smhcore# more lagttp.ttp

Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing Port Status: S -- Selected, U -- Unselected, I -- Individual, * -- Management port Flags: A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation, D -- Synchronization, E -- Collecting, F -- Distributing, G -- Defaulted, H -- Expired

Aggregate Interface: Bridge-Aggregation1 Aggregation Mode: Dynamic Loadsharing Type: Shar Management VLAN : None System ID: 0x8000, d07e-28b5-a200 Local: Port Status Priority Oper-Key Flag

GE6/0/1 U 32768 1 {ACG} GE6/0/2 U 32768 1 {ACG} Remote: Actor Partner Priority Oper-Key SystemID Flag

GE6/0/1 0 32768 0 0x8000, 0000-0000-0000 {EF} GE6/0/2 0 32768 0 0x8000, 0000-0000-0000 {EF}

Aggregate Interface: Bridge-Aggregation2 Aggregation Mode: Dynamic Loadsharing Type: Shar Management VLAN : None System ID: 0x8000, d07e-28b5-a200 Local: Port Status Priority Oper-Key Flag

GE6/0/3 U 32768 2 {ACG} GE6/0/4 U 32768 2 {ACG} Remote: Actor Partner Priority Oper-Key SystemID Flag

GE6/0/3 0 32768 0 0x8000, 0000-0000-0000 {EF} GE6/0/4 0 32768 0 0x8000, 0000-0000-0000 {EF}

Aggregate Interface: Bridge-Aggregation{{ channel_number}} Local: {{_start_}} {{ ignore(" Port.*") }} {{ ignore("-+") }} {{interface}} {{status}} {{priority}} {{oper_key}} {{flag}} Remote: {{ _start_ }} {{ ignore(" Port.*")}} {{ ignore("-+")}} {{interface}} {{status}} {{priority}} {{oper_key}} {{sys_id}}, {{ mac }} {{flag}}

@.***:/home/admin1/ttp/smhcore# ttp -t lagttp.ttp -o yaml

dmulyalin commented 3 years ago

Glad to hear that TTP helps out.

Would this work better for you:

<group name = "local_members*">
Local: {{ _start_ }}
  {{ interface }} {{ status }} {{ priority }} {{ oper_key }} {{ flag }}
</group>

What method="table" doing it is implicitly adding {{ _start_ }} to each line in the group, not sure but looks like this is not what you need.

Also no need to add {{ ignore }} for each line you want to ignore, if line not matched by any of regexes, TTP ignores it anyway.

iskobkarev commented 3 years ago

Yes I tried that but without the "table" it gets no matches at all - see attached ("local_members" group is empty)

As every group match line gets an implicit start I also need to ignore otherwise I'm getting an entry with the headers line. On Thu, Oct 14, 2021 at 11:40 PM dmulyalin ***@***.***> wrote: > Glad to hear that TTP helps out. > > Would this work better for you: > > > Local: {{ _start_ }} > {{ interface }} {{ status }} {{ priority }} {{ oper_key }} {{ flag }} >

What method="table" doing it is implicitly adding {{ start }} to each line in the group, not sure but looks like this is not what you need.

Also no need to add {{ ignore }} for each line you want to ignore, if line not matched by any of regexes, TTP ignores it anyway.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmulyalin/ttp/issues/57#issuecomment-944041029, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATJNEUZ2WEBZEZS3DGS3S7DUG7EHHANCNFSM5EDGLZDA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Thanks,

Igor Skobkarev, @.***

@.***:/home/admin1/ttp/smhcore# ttp -t lagttp.ttp -o yaml

    • portchannel: '1': local_members:
      • {} remote_members:
      • flag: '{EF}' interface: GE6/0/1 mac: 0000-0000-0000 oper_key: '0' priority: '32768' status: '0' sys_id: '0x8000'
      • flag: '{EF}' interface: GE6/0/2 mac: 0000-0000-0000 oper_key: '0' priority: '32768' status: '0' sys_id: '0x8000' '2': local_members:
      • {} remote_members:
      • flag: '{EF}' interface: GE6/0/3 mac: 0000-0000-0000 oper_key: '0' priority: '32768' status: '0' sys_id: '0x8000'
      • flag: '{EF}' interface: GE6/0/4 mac: 0000-0000-0000 oper_key: '0' priority: '32768' status: '0' sys_id: '0x8000'

@.:/home/admin1/ttp/smhcore# nano lagttp.ttp @.:/home/admin1/ttp/smhcore# nano lagttp.ttp @.***:/home/admin1/ttp/smhcore# more lagttp.ttp

Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing Port Status: S -- Selected, U -- Unselected, I -- Individual, * -- Management port Flags: A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation, D -- Synchronization, E -- Collecting, F -- Distributing, G -- Defaulted, H -- Expired

Aggregate Interface: Bridge-Aggregation1 Aggregation Mode: Dynamic Loadsharing Type: Shar Management VLAN : None System ID: 0x8000, d07e-28b5-a200 Local: Port Status Priority Oper-Key Flag

GE6/0/1 U 32768 1 {ACG} GE6/0/2 U 32768 1 {ACG} Remote: Actor Partner Priority Oper-Key SystemID Flag

GE6/0/1 0 32768 0 0x8000, 0000-0000-0000 {EF} GE6/0/2 0 32768 0 0x8000, 0000-0000-0000 {EF}

Aggregate Interface: Bridge-Aggregation2 Aggregation Mode: Dynamic Loadsharing Type: Shar Management VLAN : None System ID: 0x8000, d07e-28b5-a200 Local: Port Status Priority Oper-Key Flag

GE6/0/3 U 32768 2 {ACG} GE6/0/4 U 32768 2 {ACG} Remote: Actor Partner Priority Oper-Key SystemID Flag

GE6/0/3 0 32768 0 0x8000, 0000-0000-0000 {EF} GE6/0/4 0 32768 0 0x8000, 0000-0000-0000 {EF}

Aggregate Interface: Bridge-Aggregation{{ channel_number}} Local: {{_start_}} {{interface}} {{status}} {{priority}} {{oper_key}} {{flag}} Remote: {{ _start_ }} {{ ignore(" Port.*")}} {{ ignore("-+")}} {{interface}} {{status}} {{priority}} {{oper_key}} {{sys_id}}, {{ mac }} {{flag}}

@.***:/home/admin1/ttp/smhcore# ttp -t lagttp.ttp -o yaml

    • portchannel: '1': local_members:
      • {} remote_members:
      • flag: '{EF}' interface: GE6/0/1 mac: 0000-0000-0000 oper_key: '0' priority: '32768' status: '0' sys_id: '0x8000'
      • flag: '{EF}' interface: GE6/0/2 mac: 0000-0000-0000 oper_key: '0' priority: '32768' status: '0' sys_id: '0x8000' '2': local_members:
      • {} remote_members:
      • flag: '{EF}' interface: GE6/0/3 mac: 0000-0000-0000 oper_key: '0' priority: '32768' status: '0' sys_id: '0x8000'
      • flag: '{EF}' interface: GE6/0/4 mac: 0000-0000-0000 oper_key: '0' priority: '32768' status: '0' sys_id: '0x8000'
dmulyalin commented 3 years ago

Try to remove ignore statements and use exclude to filter results.

<group name = "portchannel.{{channel_number}}">
Aggregate Interface: Bridge-Aggregation{{ channel_number}}

<group name = "local_members*">
Local: {{_start_}}
  <group>
  {{interface}} {{status}} {{priority}} {{oper_key}} {{flag}}
  </group>
</group>

<group name = "remote_members*">
Remote: {{ _start_ }}
 <group>
  {{ interface | exclude("Port") | exclude("--") }} {{status}} {{priority}} {{oper_key}} {{sys_id}}, {{ mac }} {{flag}}
 </group>
</group>

</group>

May as well use other filter functions to filter results - https://ttp.readthedocs.io/en/latest/Match%20Variables/Functions.html#id3

Will try to test above template as well later on.

iskobkarev commented 3 years ago

thanks for the "exclude" reference -- but the output still contains an empty element and I can't figure out what creates it. It disappears when I remove "*" after the group name but then there will be no list for a single member.

See a data sample here attached.

Thx

Igor.

On Fri, Oct 15, 2021 at 4:26 PM dmulyalin @.***> wrote:

Try to remove ignore statements and use exclude to filter results.

Aggregate Interface: Bridge-Aggregation{{ channel_number}} Local: {{_start_}} {{interface}} {{status}} {{priority}} {{oper_key}} {{flag}} Remote: {{ _start_ }} {{ interface | exclude("Port") | exclude("--") }} {{status}} {{priority}} {{oper_key}} {{sys_id}}, {{ mac }} {{flag}}

May as well use other filter functions to filter results - https://ttp.readthedocs.io/en/latest/Match%20Variables/Functions.html#id3

Also could you share data sample with indentation preserved using code block or attach as a text file, will try to test above template as well.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmulyalin/ttp/issues/57#issuecomment-944805109, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATJNEU57SQR5FGJXAEEQQBLUHC2D5ANCNFSM5EDGLZDA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Thanks,

Igor Skobkarev, @.***

Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing Port Status: S -- Selected, U -- Unselected, I -- Individual, * -- Management port Flags: A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation, D -- Synchronization, E -- Collecting, F -- Distributing, G -- Defaulted, H -- Expired

Aggregate Interface: Bridge-Aggregation1 Aggregation Mode: Dynamic Loadsharing Type: Shar Management VLAN : None System ID: 0x8000, d07e-28b5-a200 Local: Port Status Priority Oper-Key Flag

GE6/0/1 U 32768 1 {ACG} GE6/0/2 U 32768 1 {ACG} Remote: Actor Partner Priority Oper-Key SystemID Flag

GE6/0/1 0 32768 0 0x8000, 0000-0000-0000 {EF} GE6/0/2 0 32768 0 0x8000, 0000-0000-0000 {EF}

Aggregate Interface: Bridge-Aggregation2 Aggregation Mode: Dynamic Loadsharing Type: Shar Management VLAN : None System ID: 0x8000, d07e-28b5-a200 Local: Port Status Priority Oper-Key Flag

GE6/0/3 U 32768 2 {ACG} GE6/0/4 U 32768 2 {ACG} Remote: Actor Partner Priority Oper-Key SystemID Flag

GE6/0/3 0 32768 0 0x8000, 0000-0000-0000 {EF} GE6/0/4 0 32768 0 0x8000, 0000-0000-0000 {EF}

iskobkarev commented 3 years ago

did more digging.

I've tried "exclude" for all variables the local_members group and that produces an empty local_members dict

remote_member does not produce a match for the headers line as there one column is actually split into two values so the headers line does not produce a match there. The exclude is not needed.

On Fri, Oct 15, 2021 at 9:24 PM Igor Skobkarev @.***> wrote:

thanks for the "exclude" reference -- but the output still contains an empty element and I can't figure out what creates it. It disappears when I remove "*" after the group name but then there will be no list for a single member.

See a data sample here attached.

Thx

Igor.

On Fri, Oct 15, 2021 at 4:26 PM dmulyalin @.***> wrote:

Try to remove ignore statements and use exclude to filter results.

Aggregate Interface: Bridge-Aggregation{{ channel_number}} Local: {{_start_}} {{interface}} {{status}} {{priority}} {{oper_key}} {{flag}} Remote: {{ _start_ }} {{ interface | exclude("Port") | exclude("--") }} {{status}} {{priority}} {{oper_key}} {{sys_id}}, {{ mac }} {{flag}}

May as well use other filter functions to filter results - https://ttp.readthedocs.io/en/latest/Match%20Variables/Functions.html#id3

Also could you share data sample with indentation preserved using code block or attach as a text file, will try to test above template as well.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmulyalin/ttp/issues/57#issuecomment-944805109, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATJNEU57SQR5FGJXAEEQQBLUHC2D5ANCNFSM5EDGLZDA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Thanks,

Igor Skobkarev, @.***

-- Thanks,

Igor Skobkarev, @.***

dmulyalin commented 3 years ago

Try this template:

<group name = "portchannel.{{channel_number}}">
Aggregate Interface: Bridge-Aggregation{{ channel_number}}

<group name = "local_members*">
  {{interface}} {{status}} {{priority | DIGIT}} {{oper_key | DIGIT}} {{flag}}
</group>

<group name = "remote_members*">
  {{interface}} {{status}} {{priority | DIGIT}} {{oper_key | DIGIT}} {{sys_id}}, {{ mac | MAC }} {{flag}}
</group>

</group>

For this data:

Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing
Port Status: S -- Selected, U -- Unselected,
             I -- Individual, * -- Management port
Flags:  A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation,
        D -- Synchronization, E -- Collecting, F -- Distributing,
        G -- Defaulted, H -- Expired

Aggregate Interface: Bridge-Aggregation1
Aggregation Mode: Dynamic
Loadsharing Type: Shar
Management VLAN : None
System ID: 0x8000, d07e-28b5-a200
Local:
  Port             Status  Priority Oper-Key  Flag
--------------------------------------------------------------------------------
  GE6/0/1          U       32768    1         {ACG}
  GE6/0/2          U       32768    1         {ACG}
Remote:
  Actor            Partner Priority Oper-Key  SystemID               Flag
--------------------------------------------------------------------------------
  GE6/0/1          0       32768    0         0x8000, 0000-0000-0000 {EF}
  GE6/0/2          0       32768    0         0x8000, 0000-0000-0000 {EF}

Aggregate Interface: Bridge-Aggregation2
Aggregation Mode: Dynamic
Loadsharing Type: Shar
Management VLAN : None
System ID: 0x8000, d07e-28b5-a200
Local:
  Port             Status  Priority Oper-Key  Flag
--------------------------------------------------------------------------------
  GE6/0/3          U       32768    2         {ACG}
  GE6/0/4          U       32768    2         {ACG}
Remote:
  Actor            Partner Priority Oper-Key  SystemID               Flag
--------------------------------------------------------------------------------
  GE6/0/3          0       32768    0         0x8000, 0000-0000-0000 {EF}
  GE6/0/4          0       32768    0         0x8000, 0000-0000-0000 {EF}

It gives these results:

[[{'portchannel': {'1': {'local_members': [{'flag': '{ACG}',
                                            'interface': 'GE6/0/1',
                                            'oper_key': '1',
                                            'priority': '32768',
                                            'status': 'U'},
                                           {'flag': '{ACG}',
                                            'interface': 'GE6/0/2',
                                            'oper_key': '1',
                                            'priority': '32768',
                                            'status': 'U'}],
                         'remote_members': [{'flag': '{EF}',
                                             'interface': 'GE6/0/1',
                                             'mac': '0000-0000-0000',
                                             'oper_key': '0',
                                             'priority': '32768',
                                             'status': '0',
                                             'sys_id': '0x8000'},
                                            {'flag': '{EF}',
                                             'interface': 'GE6/0/2',
                                             'mac': '0000-0000-0000',
                                             'oper_key': '0',
                                             'priority': '32768',
                                             'status': '0',
                                             'sys_id': '0x8000'}]},
                   '2': {'local_members': [{'flag': '{ACG}',
                                            'interface': 'GE6/0/3',
                                            'oper_key': '2',
                                            'priority': '32768',
                                            'status': 'U'},
                                           {'flag': '{ACG}',
                                            'interface': 'GE6/0/4',
                                            'oper_key': '2',
                                            'priority': '32768',
                                            'status': 'U'}],
                         'remote_members': [{'flag': '{EF}',
                                             'interface': 'GE6/0/3',
                                             'mac': '0000-0000-0000',
                                             'oper_key': '0',
                                             'priority': '32768',
                                             'status': '0',
                                             'sys_id': '0x8000'},
                                            {'flag': '{EF}',
                                             'interface': 'GE6/0/4',
                                             'mac': '0000-0000-0000',
                                             'oper_key': '0',
                                             'priority': '32768',
                                             'status': '0',
                                             'sys_id': '0x8000'}]}}}]]

If your regexes are specific enough, no need to indicate start or stop of the group using Local: {{_start_}} or Remote: {{ _start_ }}, regex formatters DIGIT and MAC allows tells TTP to use more specific regexes filtering unwanted matches.

If you need to use Local: {{_start_}} and Remote: {{ _start_ }} to indicate start of the group, consider this template:

<group name = "portchannel.{{channel_number}}">
Aggregate Interface: Bridge-Aggregation{{ channel_number}}

<group name = "local_members*" void="">
Local: {{_start_}}
  <group>
  {{interface  }} {{status}} {{priority | DIGIT }} {{oper_key | DIGIT }} {{flag}}
  </group>
</group>

<group name = "remote_members*">
Remote: {{ _start_ }}
  <group>
  {{interface }} {{status}} {{priority}} {{oper_key}} {{sys_id}}, {{ mac | MAC }} {{flag}}
  </group>
</group>

</group>

it should give same results, using void="" allows to filter get rid of empty matches {} which appear due to bug in results saving logic for ...* list path formatter - void tells to delete results for parent group which were empty anyway, fixed that behaviour in commit dc58fd435e87bddd4b722250987ef1da09ea7a8a and latest code does not require to use void=""

iskobkarev commented 3 years ago

thank you, that is one very informative response. I have started using more specific patterns (like a "connected|notconnect|errdisabled|routed" pattern for interface state instead of a plain string match) in other templates. I'll try the new commit after I'm done with my switch migration next week to avoid any unexpected changes in ttp behavior.

Are you planning to do a new formal release any time soon? Also, would it make sense to introduce a ttp version/commit variable available/output-able in the templates?

Thx

On Sun, Oct 17, 2021 at 4:20 AM dmulyalin @.***> wrote:

Try this template:

Aggregate Interface: Bridge-Aggregation{{ channel_number}} {{interface}} {{status}} {{priority | DIGIT}} {{oper_key | DIGIT}} {{flag}} {{interface}} {{status}} {{priority | DIGIT}} {{oper_key | DIGIT}} {{sys_id}}, {{ mac | MAC }} {{flag}}

For this data:

Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing Port Status: S -- Selected, U -- Unselected, I -- Individual, * -- Management port Flags: A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation, D -- Synchronization, E -- Collecting, F -- Distributing, G -- Defaulted, H -- Expired

Aggregate Interface: Bridge-Aggregation1 Aggregation Mode: Dynamic Loadsharing Type: Shar Management VLAN : None System ID: 0x8000, d07e-28b5-a200 Local: Port Status Priority Oper-Key Flag

GE6/0/1 U 32768 1 {ACG} GE6/0/2 U 32768 1 {ACG} Remote: Actor Partner Priority Oper-Key SystemID Flag

GE6/0/1 0 32768 0 0x8000, 0000-0000-0000 {EF} GE6/0/2 0 32768 0 0x8000, 0000-0000-0000 {EF}

Aggregate Interface: Bridge-Aggregation2 Aggregation Mode: Dynamic Loadsharing Type: Shar Management VLAN : None System ID: 0x8000, d07e-28b5-a200 Local: Port Status Priority Oper-Key Flag

GE6/0/3 U 32768 2 {ACG} GE6/0/4 U 32768 2 {ACG} Remote: Actor Partner Priority Oper-Key SystemID Flag

GE6/0/3 0 32768 0 0x8000, 0000-0000-0000 {EF} GE6/0/4 0 32768 0 0x8000, 0000-0000-0000 {EF}

It gives these results:

[[{'portchannel': {'1': {'local_members': [{'flag': '{ACG}', 'interface': 'GE6/0/1', 'oper_key': '1', 'priority': '32768', 'status': 'U'}, {'flag': '{ACG}', 'interface': 'GE6/0/2', 'oper_key': '1', 'priority': '32768', 'status': 'U'}], 'remote_members': [{'flag': '{EF}', 'interface': 'GE6/0/1', 'mac': '0000-0000-0000', 'oper_key': '0', 'priority': '32768', 'status': '0', 'sys_id': '0x8000'}, {'flag': '{EF}', 'interface': 'GE6/0/2', 'mac': '0000-0000-0000', 'oper_key': '0', 'priority': '32768', 'status': '0', 'sys_id': '0x8000'}]}, '2': {'local_members': [{'flag': '{ACG}', 'interface': 'GE6/0/3', 'oper_key': '2', 'priority': '32768', 'status': 'U'}, {'flag': '{ACG}', 'interface': 'GE6/0/4', 'oper_key': '2', 'priority': '32768', 'status': 'U'}], 'remote_members': [{'flag': '{EF}', 'interface': 'GE6/0/3', 'mac': '0000-0000-0000', 'oper_key': '0', 'priority': '32768', 'status': '0', 'sys_id': '0x8000'}, {'flag': '{EF}', 'interface': 'GE6/0/4', 'mac': '0000-0000-0000', 'oper_key': '0', 'priority': '32768', 'status': '0', 'sys_id': '0x8000'}]}}}]]

If your regexes are specific enough, no need to indicate start or stop of the group using Local: {{start}} or Remote: {{ start }}, regex formatters DIGIT and MAC allows tells TTP to use more specific regexes filtering unwanted matches.

If you need to use Local: {{start}} and Remote: {{ start }} to indicate start of the group, consider this template:

Aggregate Interface: Bridge-Aggregation{{ channel_number}} Local: {{_start_}} {{interface }} {{status}} {{priority | DIGIT }} {{oper_key | DIGIT }} {{flag}} Remote: {{ _start_ }} {{interface }} {{status}} {{priority}} {{oper_key}} {{sys_id}}, {{ mac | MAC }} {{flag}}

it should give same results, using void="" allows to filter get rid of empty matches {} which appear due to bug in results saving logic for ...* list path formatter - void tells to delete results for parent group which were empty anyway, fixed that behaviour in commit dc58fd4 https://github.com/dmulyalin/ttp/commit/dc58fd435e87bddd4b722250987ef1da09ea7a8a and latest code does not require to use void=""

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmulyalin/ttp/issues/57#issuecomment-945094981, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATJNEU5LJ6LYNBRIJHVPTQDUHKWP3ANCNFSM5EDGLZDA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Thanks,

Igor Skobkarev, @.***

dmulyalin commented 3 years ago

Released 0.8.1 yesterday, it includes fix for the bug I described above, its on PyPI, feel free to give it go.

ttp version - you mean something like this

<vars>
ttp_version="getttpversion"
</vars>

<group name="interfaces">
interface {{ interface }}
 description {{ description }}
 {{ ttpversion | set("ttp_version") }}
</group>

wonder, what would be the use cases for that?

If you need to do it now, you can get version of ttp using setuptools (pip install setuptools)

from pkg_resources import working_set

for pkg in working_set:
    if pkg.project_name.lower() == "ttp":
        ttp_version = pkg.version
        break
iskobkarev commented 3 years ago

Hi Denis

Came across one more odd issue - the data and the script are in the attachment.

it looks like joinmatches() does not work as expected. That data/script below does not collapse the dhcp relays under one variable. Am I missing something?

@.:/home/admin1/ttp/kdcconv# more cmw5-dhcp.cfg dhcp relay server-group 1 ip 10.248.217.1 dhcp relay server-group 1 ip 10.248.217.2 dhcp relay server-group 3 ip 10.170.100.51 dhcp relay server-group 3 ip 10.170.100.52 dhcp relay server-detect # @.:/home/admin1/ttp/kdcconv# more t.ttp

DHCP

dhcp relay server-group {{group_number}} ip {{dhcp_relay | joinmatches(",")}}

On Sun, Oct 17, 2021 at 2:41 PM dmulyalin @.***> wrote:

Released 0.8.1 yesterday, it includes fix for the bug I described above, its on PyPI, feel free to give it go.

ttp version - you mean something like this

ttp_version="getttpversion" interface {{ interface }} description {{ description }} {{ ttpversion | set("ttp_version") }}

wonder, what would be the use cases for that?

If you need to do it now, you can get version of ttp using setuptools (pip install setuptools)

from pkg_resources import working_set

for pkg in working_set: if pkg.project_name.lower() == "ttp": ttp_version = pkg.version break

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmulyalin/ttp/issues/57#issuecomment-945199589, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATJNEU6BCHQTJT3OUFW6CR3UHM7KNANCNFSM5EDGLZDA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Thanks,

Igor Skobkarev, @.***

@.:/home/admin1/ttp/kdcconv# more cmw5-dhcp.cfg dhcp relay server-group 1 ip 10.248.217.1 dhcp relay server-group 1 ip 10.248.217.2 dhcp relay server-group 3 ip 10.170.100.51 dhcp relay server-group 3 ip 10.170.100.52 dhcp relay server-detect # @.:/home/admin1/ttp/kdcconv# more t.ttp

DHCP

dhcp relay server-group {{group_number}} ip {{dhcp_relay | joinmatches(",")}}

@.***:/home/admin1/ttp/kdcconv# ttp -d cmw5-dhcp.cfg -t t.ttp -o json [ [ { "dhcp_relay_1": [ { "dhcp_relay": "10.248.217.1" }, { "dhcp_relay": "10.248.217.2" } ], "dhcp_relay_3": [ { "dhcp_relay": "10.170.100.51" }, { "dhcp_relay": "10.170.100.52" } ] } ] ]

dmulyalin commented 3 years ago

Hi,

Yeah, there is a restriction - https://ttp.readthedocs.io/en/latest/Match%20Variables/Functions.html#joinmatches image

Workaround could be to post-process your results to form a structure you need.

iskobkarev commented 3 years ago

yes I tried to add joinmatches to both variables but the outcome was the same. Even the joinmatch example on the same page [image: image.png]

produces no joined vars if "interface" line is removed from the source data and the template [image: image.png]

I guess I will need to look into post-processing option

On Thu, Oct 28, 2021 at 2:41 PM dmulyalin @.***> wrote:

Hi,

Yeah, there is a restriction - https://ttp.readthedocs.io/en/latest/Match%20Variables/Functions.html#joinmatches [image: image] https://user-images.githubusercontent.com/28857266/139340247-970d652b-8b96-4b41-b7d2-3d887c3fa7f8.png

Workaround could be to post-process your results to form a structure you need.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmulyalin/ttp/issues/57#issuecomment-954244391, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATJNEUZCKJMYP2GLPPJXZZDUJG7QHANCNFSM5EDGLZDA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Thanks,

Igor Skobkarev, @.***

iskobkarev commented 3 years ago

I would like to support this project in some way but I could not find "donate" button anywhere...

HPE comware is giving me headache when I'm trying to parse very elaborated (a few hundred K) configurations. BGP is another example here where the bgp peering configuration is in two different spots:

Here is the data and the script

ipv4-family vpn-instance FHA router-id 10.251.0.15 peer 10.251.64.113 as-number 64991 peer 10.251.64.97 as-number 64991 aggregate 10.251.64.0 255.255.248.0 detail-suppressed attribute-policy IGP_into_FHA aggregate 10.251.160.0 255.255.248.0 detail-suppressed attribute-policy IGP_into_FHA preference 75 80 90 network 10.248.128.0 255.255.255.0 route-policy IGP_into_FHA network 10.248.137.160 255.255.255.224 route-policy IGP_into_FHA peer 10.251.64.97 route-policy fhadmz_to_fha import peer 10.251.64.97 ebgp-max-hop 5 peer 10.251.64.97 timer keepalive 1 hold 5 peer 10.251.64.97 route-update-interval 1 peer 10.251.64.97 connect-interface LoopBack101 peer 10.251.64.97 fake-as 64881 peer 10.251.64.113 route-policy fha_to_fhaimz export peer 10.251.64.113 route-policy fhaimz_to_fha import peer 10.251.64.113 ebgp-max-hop 5 peer 10.251.64.113 timer keepalive 1 hold 5 peer 10.251.64.113 route-update-interval 1 peer 10.251.64.113 connect-interface LoopBack101 peer 10.251.64.113 fake-as 64881

{{address_family}}-family vpn-instance {{vpn_instance | start}} router-id {{router_id}}

peer {{peer_id}} as-number {{peer_as_number | _start_}} peer {{peer_id}} group {{peer_group_id }} peer {{peer_id}} label_route_capability {{label_router_capability | re("label-route-capability")}} peer {{peer_id}} {{bfd | re("bfd") | let("enabled") }} peer {{peer_id}} route-policy {{peer_import_policy}} import peer {{peer_id}} route-policy {{peer_export_policy}} export peer {{peer_id}} ebgp-max-hop {{ebgp_max_hop}} peer {{peer_id}} timer keepalive {{timet_keepalive}} hold {{timer_hold}} peer {{peer_id}} route-update-interval {{timer_route_update}} peer {{peer_id}} connect-interface {{source_interface}} peer {{peer_id}} fake-as {{local_as}} aggregate {{network}} {{mask}} {{detail_suppressed }} attribute-policy {{attribute_policy}}

preference {{pref_internal}} {{pref_external}} {{pref_local}}

network {{network}} {{mask}} route-policy {{route_policy}}

The end result is that it properly gets one of the peers but not the other one (.113). One I put all the peer statements next to each other it seem to work. What would be the best approach to parse it with ttp?

[image: image.png]

On Thu, Oct 28, 2021 at 3:49 PM Igor Skobkarev @.***> wrote:

yes I tried to add joinmatches to both variables but the outcome was the same. Even the joinmatch example on the same page [image: image.png]

produces no joined vars if "interface" line is removed from the source data and the template [image: image.png]

I guess I will need to look into post-processing option

On Thu, Oct 28, 2021 at 2:41 PM dmulyalin @.***> wrote:

Hi,

Yeah, there is a restriction - https://ttp.readthedocs.io/en/latest/Match%20Variables/Functions.html#joinmatches [image: image] https://user-images.githubusercontent.com/28857266/139340247-970d652b-8b96-4b41-b7d2-3d887c3fa7f8.png

Workaround could be to post-process your results to form a structure you need.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dmulyalin/ttp/issues/57#issuecomment-954244391, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATJNEUZCKJMYP2GLPPJXZZDUJG7QHANCNFSM5EDGLZDA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Thanks,

Igor Skobkarev, @.***

-- Thanks,

Igor Skobkarev, @.***

dmulyalin commented 2 years ago

@iskobkarev

Sorry for long no reply, for HPE like cases usually quiet helpful to use dictionary structure where peer IP addresses uses as keys, in that case TTP will merge all configuration for certain peer based on its IP address, but looks like you already figured out that trick by yourself.

However, your template need to be adjusted slightly to achieve what you need:

<group name= "bgp_af.{{vpn_instance}}.{{address_family}}">
 {{address_family}}-family vpn-instance {{vpn_instance}}
  router-id {{router_id}}
  preference {{pref_internal}} {{pref_external}} {{pref_local}}

 <group name= "bgp_peer.{{peer_id}}**" method="table">
  peer {{peer_id}} as-number {{peer_as_number}}
  peer {{peer_id}} group {{peer_group_id }}
  peer {{peer_id}} label_route_capability {{label_router_capability | re("label-route-capability")}}
  peer {{peer_id}} {{bfd | re("bfd") | let("enabled") }}
  peer {{peer_id}} route-policy {{peer_import_policy}} import
  peer {{peer_id}} route-policy {{peer_export_policy}} export
  peer {{peer_id}} ebgp-max-hop {{ebgp_max_hop}}
  peer {{peer_id}} timer keepalive {{timet_keepalive}} hold {{timer_hold}}
  peer {{peer_id}} route-update-interval {{timer_route_update}}
  peer {{peer_id}} connect-interface {{source_interface}}
  peer {{peer_id}} fake-as {{local_as}}
 </group>

<group name="aggregate">
  aggregate {{network}} {{mask}} {{detail_suppressed }} attribute-policy {{attribute_policy}}
</group>

<group name="networks">
  network {{network}} {{mask}} route-policy {{route_policy}}
</group>  

</group>

Where key change is - group name= "bgp_peer.{{peer_id}}**" method="table" - method table tells that each line is a start line in the group, meaning results for each line must be save merged to overall results, {{peer_id}}** tells to use dictionary as a leaf under peer_id.

Above template produces this result:

[[{'bgp_af': {'FHA': {'ipv4': {'aggregate': [{'attribute_policy': 'IGP_into_FHA',
                                              'detail_suppressed': 'detail-suppressed',
                                              'mask': '255.255.248.0',
                                              'network': '10.251.64.0'},
                                             {'attribute_policy': 'IGP_into_FHA',
                                              'detail_suppressed': 'detail-suppressed',
                                              'mask': '255.255.248.0',
                                              'network': '10.251.160.0'}],
                               'bgp_peer': {'10.251.64.113': {'ebgp_max_hop': '5',
                                                              'local_as': '64881',
                                                              'peer_as_number': '64991',
                                                              'peer_export_policy': 'fha_to_fhaimz',
                                                              'peer_import_policy': 'fhaimz_to_fha',
                                                              'source_interface': 'LoopBack101',
                                                              'timer_hold': '5',
                                                              'timer_route_update': '1',
                                                              'timet_keepalive': '1'},
                                            '10.251.64.97': {'ebgp_max_hop': '5',
                                                             'local_as': '64881',
                                                             'peer_as_number': '64991',
                                                             'peer_import_policy': 'fhadmz_to_fha',
                                                             'source_interface': 'LoopBack101',
                                                             'timer_hold': '5',
                                                             'timer_route_update': '1',
                                                             'timet_keepalive': '1'}},
                               'networks': [{'mask': '255.255.255.0',
                                             'network': '10.248.128.0',
                                             'route_policy': 'IGP_into_FHA'},
                                            {'mask': '255.255.255.224',
                                             'network': '10.248.137.160',
                                             'route_policy': 'IGP_into_FHA'}],
                               'pref_external': '80',
                               'pref_internal': '75',
                               'pref_local': '90',
                               'router_id': '10.251.0.15'}}}}]]

Regarding - I would like to support this project in some way but I could not find "donate" button anywhere... - there is sponsor button at the top right of the repo.

iskobkarev commented 2 years ago

Thank you Denis

I'll try your solution in a bit.

I've tried to install 0.8.2 but this is what it threw at me. It does it for every template. I rolled back to 0.8.1 and all works fine.

@.:/home/admin1/ttp/route_collector/v001# pip install ttp==0.8.1 Processing /root/.cache/pip/wheels/ec/80/e1/6696719579b4232dd4bf6de9cc0789036090777f695f5643f0/ttp-0.8.1-py2.py3-none-any.whl Installing collected packages: ttp Attempting uninstall: ttp Found existing installation: ttp 0.8.2 Uninstalling ttp-0.8.2: Successfully uninstalled ttp-0.8.2 Successfully installed ttp-0.8.1 @.:/home/admin1/ttp/route_collector/v001# ttp -t t30.ttp -o yaml

@.:/home/admin1/ttp/route_collector/v001# pip install ttp==0.8.2 Processing /root/.cache/pip/wheels/62/4e/f1/ea37f245c8fccefc6e06371e2b5728978375bc8a3a80e42b4d/ttp-0.8.2-py2.py3-none-any.whl Installing collected packages: ttp Attempting uninstall: ttp Found existing installation: ttp 0.8.1 Uninstalling ttp-0.8.1: Successfully uninstalled ttp-0.8.1 Successfully installed ttp-0.8.2 @.:/home/admin1/ttp/route_collector/v001# ttp -t t30.ttp -o yaml Traceback (most recent call last): File "/usr/local/bin/ttp", line 8, in sys.exit(cli_tool()) File "/usr/local/lib/python3.8/dist-packages/ttp/ttp.py", line 3535, in cli_tool ttp_template = ttp["utils"]"load_files"[0][1] File "/usr/local/lib/python3.8/dist-packages/ttp/ttp.py", line 67, in call self.load() File "/usr/local/lib/python3.8/dist-packages/ttp/ttp.py", line 51, in load setattr(module, "ttp", self.ttp) AttributeError: 'CachedModule' object has no attribute 'ttp' @.***:/home/admin1/ttp/route_collector/v001#

On Mon, Dec 27, 2021 at 12:51 AM dmulyalin @.***> wrote:

@iskobkarev https://github.com/iskobkarev

Sorry for long no reply, for HPE like cases usually quiet helpful to use dictionary structure where peer IP addresses uses as keys, in that case TTP will merge all configuration for certain peer based on its IP address, but looks like you already figured out that trick by yourself.

However, your template need to be adjusted slightly to achieve what you need:

{{address_family}}-family vpn-instance {{vpn_instance}} router-id {{router_id}} preference {{pref_internal}} {{pref_external}} {{pref_local}} peer {{peer_id}} as-number {{peer_as_number}} peer {{peer_id}} group {{peer_group_id }} peer {{peer_id}} label_route_capability {{label_router_capability | re("label-route-capability")}} peer {{peer_id}} {{bfd | re("bfd") | let("enabled") }} peer {{peer_id}} route-policy {{peer_import_policy}} import peer {{peer_id}} route-policy {{peer_export_policy}} export peer {{peer_id}} ebgp-max-hop {{ebgp_max_hop}} peer {{peer_id}} timer keepalive {{timet_keepalive}} hold {{timer_hold}} peer {{peer_id}} route-update-interval {{timer_route_update}} peer {{peer_id}} connect-interface {{source_interface}} peer {{peer_id}} fake-as {{local_as}} aggregate {{network}} {{mask}} {{detail_suppressed }} attribute-policy {{attribute_policy}} network {{network}} {{mask}} route-policy {{route_policy}}

Where key change is - group name= "bgp_peer.{{peer_id}}**" method="table"

  • method table tells that each line is a start line in the group, meaning results for each line must be save merged to overall results, {{peer_id}}** tells to use dictionary as a leaf under peer_id.

Above template produces this result:

[[{'bgp_af': {'FHA': {'ipv4': {'aggregate': [{'attribute_policy': 'IGP_into_FHA', 'detail_suppressed': 'detail-suppressed', 'mask': '255.255.248.0', 'network': '10.251.64.0'}, {'attribute_policy': 'IGP_into_FHA', 'detail_suppressed': 'detail-suppressed', 'mask': '255.255.248.0', 'network': '10.251.160.0'}], 'bgp_peer': {'10.251.64.113': {'ebgp_max_hop': '5', 'local_as': '64881', 'peer_as_number': '64991', 'peer_export_policy': 'fha_to_fhaimz', 'peer_import_policy': 'fhaimz_to_fha', 'source_interface': 'LoopBack101', 'timer_hold': '5', 'timer_route_update': '1', 'timet_keepalive': '1'}, '10.251.64.97': {'ebgp_max_hop': '5', 'local_as': '64881', 'peer_as_number': '64991', 'peer_import_policy': 'fhadmz_to_fha', 'source_interface': 'LoopBack101', 'timer_hold': '5', 'timer_route_update': '1', 'timet_keepalive': '1'}}, 'networks': [{'mask': '255.255.255.0', 'network': '10.248.128.0', 'route_policy': 'IGP_into_FHA'}, {'mask': '255.255.255.224', 'network': '10.248.137.160', 'route_policy': 'IGP_into_FHA'}], 'pref_external': '80', 'pref_internal': '75', 'pref_local': '90', 'router_id': '10.251.0.15'}}}}]]

Regarding - I would like to support this project in some way but I could not find "donate" button anywhere... - there is sponsor button at the top right of the repo.

— Reply to this email directly, view it on GitHub https://github.com/dmulyalin/ttp/issues/57#issuecomment-1001440631, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATJNEU3OUAC6UIANHMMX4ZTUTASKJANCNFSM5EDGLZDA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

-- Thanks,

Igor Skobkarev, @.***

dmulyalin commented 2 years ago

Thanks for raising this issue, should be fixed in 0.8.3 details of the problem described in #66.

iskobkarev commented 2 years ago

Hi Dennis

Back to that parsing puzzle. It works fine when there is a peer id acting as a key to build the result together. What about the situation when there is no such key, like in HPE route-policies - see the example below:

When I apply method="table" and use this it creates a fresh entry under the matchRules list: If I use two stars ( ) it picks just the first "if-match" and ignores the rest.

If I don't use the table the result could be great (if the first line in the output happened to be the first match in the template) or no match at all. It is almost as if it needs one more method on top of group/table ones, where any line match within the group would start the group record but would not create many.

What would be the way to attack this and similar outputs? My work in progress template is attached.

Route-policy : IGP_into_HSSBC deny : 5 if-match tag 102 if-match community commlist permit : 10 apply community 64800:1001 64800:7 additive apply as-path 1 2 3 4 5 apply cost + 10 apply cost-type type-1

On Mon, Dec 27, 2021 at 12:51 AM dmulyalin @.***> wrote:

@iskobkarev https://github.com/iskobkarev

Sorry for long no reply, for HPE like cases usually quiet helpful to use dictionary structure where peer IP addresses uses as keys, in that case TTP will merge all configuration for certain peer based on its IP address, but looks like you already figured out that trick by yourself.

However, your template need to be adjusted slightly to achieve what you need:

{{address_family}}-family vpn-instance {{vpn_instance}} router-id {{router_id}} preference {{pref_internal}} {{pref_external}} {{pref_local}} peer {{peer_id}} as-number {{peer_as_number}} peer {{peer_id}} group {{peer_group_id }} peer {{peer_id}} label_route_capability {{label_router_capability | re("label-route-capability")}} peer {{peer_id}} {{bfd | re("bfd") | let("enabled") }} peer {{peer_id}} route-policy {{peer_import_policy}} import peer {{peer_id}} route-policy {{peer_export_policy}} export peer {{peer_id}} ebgp-max-hop {{ebgp_max_hop}} peer {{peer_id}} timer keepalive {{timet_keepalive}} hold {{timer_hold}} peer {{peer_id}} route-update-interval {{timer_route_update}} peer {{peer_id}} connect-interface {{source_interface}} peer {{peer_id}} fake-as {{local_as}} aggregate {{network}} {{mask}} {{detail_suppressed }} attribute-policy {{attribute_policy}} network {{network}} {{mask}} route-policy {{route_policy}}

Where key change is - group name= "bgp_peer.{{peer_id}}**" method="table"

  • method table tells that each line is a start line in the group, meaning results for each line must be save merged to overall results, {{peer_id}}** tells to use dictionary as a leaf under peer_id.

Above template produces this result:

[[{'bgp_af': {'FHA': {'ipv4': {'aggregate': [{'attribute_policy': 'IGP_into_FHA', 'detail_suppressed': 'detail-suppressed', 'mask': '255.255.248.0', 'network': '10.251.64.0'}, {'attribute_policy': 'IGP_into_FHA', 'detail_suppressed': 'detail-suppressed', 'mask': '255.255.248.0', 'network': '10.251.160.0'}], 'bgp_peer': {'10.251.64.113': {'ebgp_max_hop': '5', 'local_as': '64881', 'peer_as_number': '64991', 'peer_export_policy': 'fha_to_fhaimz', 'peer_import_policy': 'fhaimz_to_fha', 'source_interface': 'LoopBack101', 'timer_hold': '5', 'timer_route_update': '1', 'timet_keepalive': '1'}, '10.251.64.97': {'ebgp_max_hop': '5', 'local_as': '64881', 'peer_as_number': '64991', 'peer_import_policy': 'fhadmz_to_fha', 'source_interface': 'LoopBack101', 'timer_hold': '5', 'timer_route_update': '1', 'timet_keepalive': '1'}}, 'networks': [{'mask': '255.255.255.0', 'network': '10.248.128.0', 'route_policy': 'IGP_into_FHA'}, {'mask': '255.255.255.224', 'network': '10.248.137.160', 'route_policy': 'IGP_into_FHA'}], 'pref_external': '80', 'pref_internal': '75', 'pref_local': '90', 'router_id': '10.251.0.15'}}}}]]

Regarding - I would like to support this project in some way but I could not find "donate" button anywhere... - there is sponsor button at the top right of the repo.

— Reply to this email directly, view it on GitHub https://github.com/dmulyalin/ttp/issues/57#issuecomment-1001440631, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATJNEU3OUAC6UIANHMMX4ZTUTASKJANCNFSM5EDGLZDA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

-- Thanks,

Igor Skobkarev, @.***

SCRIPT::::

Route-policy : IGP_into_HSSBC deny : 5 if-match tag 102 if-match community commlist permit : 10 if-match tag 103 if-match community permitcom apply community 64800:1001 64800:7 additive apply as-path 1 2 3 4 5 apply cost + 10 apply cost-type type-1

ACTION = "Permit|Deny|permit|deny" Route-policy : {{rp_name | _start_}} {{filterType | re("ACTION") }} : {{node}} if-match tag {{ tag | default("") }} if-match community {{ community | default("") }} if-match interface {{ interface | default("") }} if-match ip-prefix {{ ipPrefix | default("") }} if-match route-type {{ routeType | default("") }} if-match acl {{ ipAcl | default("") }} if-match cost {{ cost | default("") }} if-match extcommunity {{ extCommunity | default("") }} apply as-path {{ setAsPathReplace | ORPHRASE | default("") }} replace apply as-path {{ setAsPath | ROW | default("") }} apply community {{ setCommunityAdditive | default("") }} additive apply community {{ setCommunity | ORPHRASE | default("") }} apply cost {{ cost | ORPHRASE | default("") }} apply cost-type {{ costType | ORPHRASE | default("") }} apply tag {{ tag | ORPHRASE | default("") }}

RESULT::

dmulyalin commented 2 years ago

Hi, in relation to the latest comment, could you share sample data and template together with results you getting and expected results, will try to investigate and see what TTP can do.

iskobkarev commented 2 years ago

Hi - There was an attachment. re-sending

thx

On Tue, Feb 22, 2022 at 1:43 PM dmulyalin @.***> wrote:

Hi, in relation to the latest comment, could you share sample data and template together with results you getting and expected results, will try to investigate and see what TTP can do.

— Reply to this email directly, view it on GitHub https://github.com/dmulyalin/ttp/issues/57#issuecomment-1048240464, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATJNEU6HBA5XFSAWXGH4TFLU4P7O3ANCNFSM5EDGLZDA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

-- Thanks,

Igor Skobkarev, @.***

SCRIPT::::

Route-policy : IGP_into_HSSBC deny : 5 if-match tag 102 if-match community commlist permit : 10 if-match tag 103 if-match community permitcom apply community 64800:1001 64800:7 additive apply as-path 1 2 3 4 5 apply cost + 10 apply cost-type type-1

ACTION = "Permit|Deny|permit|deny" Route-policy : {{rp_name | _start_}} {{filterType | re("ACTION") }} : {{node}} if-match tag {{ tag | default("") }} if-match community {{ community | default("") }} if-match interface {{ interface | default("") }} if-match ip-prefix {{ ipPrefix | default("") }} if-match route-type {{ routeType | default("") }} if-match acl {{ ipAcl | default("") }} if-match cost {{ cost | default("") }} if-match extcommunity {{ extCommunity | default("") }} apply as-path {{ setAsPathReplace | ORPHRASE | default("") }} replace apply as-path {{ setAsPath | ROW | default("") }} apply community {{ setCommunityAdditive | default("") }} additive apply community {{ setCommunity | ORPHRASE | default("") }} apply cost {{ cost | ORPHRASE | default("") }} apply cost-type {{ costType | ORPHRASE | default("") }} apply tag {{ tag | ORPHRASE | default("") }}

RESULT::

dmulyalin commented 2 years ago

Maybe this template will do the trick.

Data:

Route-policy : IGP_into_HSSBC
  deny   : 5
        if-match tag 102
        if-match community commlist
  permit : 10
        if-match community permitcom
        apply community 64800:1001 64800:7 additive
        apply as-path 1 2 3 4 5
        apply cost + 10
        apply cost-type type-1
  permit : 20
        apply community 64800:1001 64800:7 additive
        apply as-path 1 2 3 4 5
        apply cost + 10
        apply cost-type type-1

Template:

<vars>
ACTION = "Permit|Deny|permit|deny"
</vars>

<group name="{{rp_name}}">
Route-policy : {{rp_name | _start_}}

<group name="{{node}}" expand="">
  {{filterType | re("ACTION") }}  : {{node}}

        if-match tag {{matchRules.tag | default("") }}
        if-match community {{ matchRules.community | default("") }}
        if-match interface {{ matchRules.interface | default("") }}
        if-match ip-prefix {{ matchRules.ipPrefix | default("")  }}
        if-match route-type {{ matchRules.routeType | default("")  }}
        if-match acl {{ matchRules.ipAcl | default("") }}
        if-match cost {{ matchRules.cost | default("") }}
        if-match extcommunity {{ matchRules.extCommunity | default("") }}

        apply as-path {{ setRules.setAsPathReplace | ORPHRASE | default("") }} replace
        apply as-path {{ setRules.setAsPath | ROW | default("") }}
        apply community {{ setRules.setCommunityAdditive | default("") }} additive
        apply community {{ setRules.setCommunity | ORPHRASE | default("") }}
        apply cost {{ setRules.cost | ORPHRASE | default("") }}
        apply cost-type {{ setRules.costType | ORPHRASE | default("") }}
        apply tag {{ setRules.tag | ORPHRASE | default("") }}

</group>
</group>

Gives this result:

[
    {
        "IGP_into_HSSBC": {
            "5": {
                "filterType": "deny",
                "matchRules": {
                    "community": "commlist",
                    "cost": "",
                    "extCommunity": "",
                    "interface": "",
                    "ipAcl": "",
                    "ipPrefix": "",
                    "routeType": "",
                    "tag": "102"
                },
                "setRules": {
                    "cost": "",
                    "costType": "",
                    "setAsPath": "",
                    "setAsPathReplace": "",
                    "setCommunity": "",
                    "setCommunityAdditive": "",
                    "tag": ""
                }
            },
            "10": {
                "filterType": "permit",
                "matchRules": {
                    "community": "permitcom",
                    "cost": "",
                    "extCommunity": "",
                    "interface": "",
                    "ipAcl": "",
                    "ipPrefix": "",
                    "routeType": "",
                    "tag": ""
                },
                "setRules": {
                    "cost": "+ 10",
                    "costType": "type-1",
                    "setAsPath": "1 2 3 4 5",
                    "setAsPathReplace": "",
                    "setCommunity": "64800:1001 64800:7 additive",
                    "setCommunityAdditive": "",
                    "tag": ""
                }
            },
            "20": {
                "filterType": "permit",
                "matchRules": {
                    "community": "",
                    "cost": "",
                    "extCommunity": "",
                    "interface": "",
                    "ipAcl": "",
                    "ipPrefix": "",
                    "routeType": "",
                    "tag": ""
                },
                "setRules": {
                    "cost": "+ 10",
                    "costType": "type-1",
                    "setAsPath": "1 2 3 4 5",
                    "setAsPathReplace": "",
                    "setCommunity": "64800:1001 64800:7 additive",
                    "setCommunityAdditive": "",
                    "tag": ""
                }
            }
        }
    }
]
JoanDuclos commented 2 years ago

Hi, sorry to bother you but this is the only way to send you a message on GitHub...

I would like to discuss with you @iskobkarev about your templates because I'm thinking about doing a similar project and it would help me a lot to move forward.