search

google / textfsm

Python module for parsing semi-structured text into python tables.
Apache License 2.0
1.11k stars 171 forks source link

How to record a Value only if it has changed? #52

Closed Geoffrey42 closed 3 years ago

Geoffrey42 commented 5 years ago

Summary

I'm parsing a CISCO BGP router address-family using TextFSM. How can I record BGP neighbor only when a new one is match ?

WARNING all below data is fake.

Some sample configuration

 address-family ipv4 vrf EASY-GOING
  no synchronization
  neighbor 10.42.80.2 remote-as 64442
  neighbor 10.42.80.2 inherit peer-session LTE-ASR
  neighbor 10.42.80.2 description *** some EASY-GOING router ***
  neighbor 10.42.80.2 activate
  neighbor 10.42.80.2 inherit peer-policy LTE-ASR
  neighbor 10.42.80.2 default-originate
 exit-address-family
 !
 address-family ipv4 vrf THE-MONTAIN
  import path selection all
  import path limit 8
  no synchronization
  redistribute static route-map advertise-primary
  redistribute connected route-map advertise-primary
  neighbor 10.144.139.165 remote-as 65021
  neighbor 10.144.139.165 description *** peering CASE DMZ MONTAINVPDMZ01 ***
  neighbor 10.144.139.165 route-map RESILIENCE_VIPS_EXT
  neighbor 10.144.139.166 remote-as 65021
  neighbor 10.144.139.166 ebgp-multihop 5
  neighbor 10.144.139.166 update-source Vlan508
  neighbor 10.144.139.166 route-map DENY_ALL out
 exit-address-family

What I want

+-------------+----------------+-----------+-----------------------------------------+---------------------+------------+-------------+
|     VRF     |    NEIGHBOR    | REMOTE-AS |               DESCRIPTION               |      ROUTE-MAP      | UPDATE-SRC | MULTIHOP_NB |
+-------------+----------------+-----------+-----------------------------------------+---------------------+------------+-------------+
| EASY-GOING  | 10.42.80.2     |     64442 | *** some EASY-GOING router ***          |                     |            |             |
| THE-MONTAIN | 10.144.139.165 |     65021 | *** peering CASE DMZ MONTAINVPDMZ01 *** | RESILIENCE_VIPS_EXT |            |             |
| THE-MONTAIN | 10.144.139.166 |           |                                         | DENY_ALL out        | Vlan508    |           5 |
+-------------+----------------+-----------+-----------------------------------------+---------------------+------------+-------------+

Warning There are far more possible configuration options for each neighbor, above output and table are just samples. Beside, there is not any pattern, a neighbor can have only one configuratoin (e.g a timer) or a very large number. The consequence is that I can't use them as separator between two different neighbors.

What I tried

Use end of block to record

$ cat templates/cisco_show_run_part_address-family_ipv4_vrf_00.template    
Value Filldown VRF (\S+)
Value NEIGHBOR (\S+)
Value REMOTE_AS (\d+)
Value DESCRIPTION (\S+)
Value ROUTE_MAP (\S+)
Value UPDATE-SRC (\S+)
Value MULTIHOP_NB (\d+)

Start
  ^ address-family ipv4 vrf ${VRF} -> NEIGHBOR

NEIGHBOR
  ^  neighbor ${NEIGHBOR} remote-as ${REMOTE_AS}
  ^  neighbor ${NEIGHBOR} route-map ${ROUTE_MAP}
  ^  neighbor ${NEIGHBOR} description ${DESCRIPTION}
  ^  neighbor ${NEIGHBOR} update-source ${UPDATE-SRC}
  ^  neighbor ${NEIGHBOR} ebgp-multihop ${MULTIHOP_NB}
  ^ exit-address-family -> Record Start
$

What I get

+-------------+----------------+-----------+-----------------------------------------+---------------------+------------+-------------+
|     VRF     |    NEIGHBOR    | REMOTE-AS |               DESCRIPTION               |      ROUTE-MAP      | UPDATE-SRC | MULTIHOP_NB |
+-------------+----------------+-----------+-----------------------------------------+---------------------+------------+-------------+
| EASY-GOING  | 10.42.80.2     |     64442 | *** some EASY-GOING router ***          |                     |            |             |
| THE-MONTAIN | 10.144.139.166 |    65021  |  *** peering CASE DMZ MONTAINVPDMZ01 ***| DENY_ALL out        | Vlan508    |           5 |
+-------------+----------------+-----------+-----------------------------------------+---------------------+------------+-------------+

Unfortunately, doing this records only the last neighbor matched and "erase" the previous one matched between two records.

Record after each neighbor line

$ cat templates/cisco_show_run_part_address-family_ipv4_vrf_01.template
Value Filldown VRF (\S+)
Value NEIGHBOR (\S+)
Value REMOTE_AS (\d+)
Value DESCRIPTION (\S+)
Value ROUTE_MAP (\S+)
Value UPDATE-SRC (\S+)
Value MULTIHOP_NB (\d+)

Start
  ^ address-family ipv4 vrf ${VRF} -> NEIGHBOR

NEIGHBOR
  ^  neighbor ${NEIGHBOR} remote-as ${REMOTE_AS} -> Record
  ^  neighbor ${NEIGHBOR} route-map ${ROUTE_MAP} -> Record
  ^  neighbor ${NEIGHBOR} description ${DESCRIPTION} -> Record
  ^  neighbor ${NEIGHBOR} update-source ${UPDATE-SRC} -> Record
  ^  neighbor ${NEIGHBOR} ebgp-multihop ${MULTIHOP_NB} -> Record
  ^ exit-address-family -> Start
$

What I get

+-------------+----------------+-----------+-----------------------------------------+---------------------+------------+-------------+
|     VRF     |    NEIGHBOR    | REMOTE-AS |               DESCRIPTION               |      ROUTE-MAP      | UPDATE-SRC | MULTIHOP_NB |
+-------------+----------------+-----------+-----------------------------------------+---------------------+------------+-------------+
| EASY-GOING  | 10.42.80.2     |     64442 |                                         |                     |            |             |
| EASY-GOING  | 10.42.80.2     |           | *** some EASY-GOING router ***          |                     |            |             |
| THE-MONTAIN | 10.144.139.165 |     65021 |                                         |                     |            |             |
| THE-MONTAIN | 10.144.139.165 |           | *** peering CASE DMZ MONTAINVPDMZ01 *** |                     |            |             |
| THE-MONTAIN | 10.144.139.165 |           |                                         | RESILIENCE_VIPS_EXT |            |             |
| THE-MONTAIN | 10.144.139.166 |           |                                         | DENY_ALL out        |            |             |
| THE-MONTAIN | 10.144.139.166 |           |                                         |                     | Vlan508    |             |
| THE-MONTAIN | 10.144.139.166 |           |                                         |                     |            |           5 |
+-------------+----------------+-----------+-----------------------------------------+---------------------+------------+-------------+

This one create a new line for each configuration on a specific neighbor. At least I can see the different neighbors for an address-family but since I'm gonna perform a lot of request (using SQL) on each of those tables, I want them to avoid unecessary duplicates. Conceptually, there shouldn't have several lines for one neighbor.

Both solutions aren't satisfying at all.

Geoffrey42 commented 5 years ago

If it's not possible, could it be possible to make a PR on it ? I get that fine tunning data formatting is not the main purpose of TextFSM but it is not the first time I encounter this issue. I never figured it out and last time I just made my own parser to overcome the problem.

The thing is I like textFSM and I want to automate a lot of routers parsing using it, so if it's only a lack of understanding/skills on textFSM, feel free to explain me or at least maybe I could add some Option (e.g Target or whatever) for a given Value to skip Record unless a new Value's value is matched.

What do you think ?

jrschneider commented 5 years ago

I was looking up a similar issue and yours intrigued me. How about this?

fh = StringIO(
"""Value Filldown VRF (\S+)
Value Required NEIGHBOR (\S+)
Value REMOTE_AS (\d+)
Value DESCRIPTION (.*)
Value ROUTE_MAP (\S+)
Value UPDATE_SRC (\S+)
Value MULTIHOP_NB (\d+)

Start
  ^ address-family ipv4 vrf ${VRF} -> NEIGHBORSTART

NEIGHBORSTART
  ^  neighbor ${NEIGHBOR} remote-as ${REMOTE_AS} -> NEIGHBOR

NEIGHBOR
  ^  neighbor ${NEIGHBOR} route-map ${ROUTE_MAP}
  ^  neighbor ${NEIGHBOR} description ${DESCRIPTION}
  ^  neighbor ${NEIGHBOR} update-source ${UPDATE_SRC}
  ^  neighbor ${NEIGHBOR} ebgp-multihop ${MULTIHOP_NB}
  ^  neighbor \S+ remote-as \d+ -> Continue.Record
  ^  neighbor ${NEIGHBOR} remote-as ${REMOTE_AS} 
  ^ exit-address-family -> Record Start""")

parser = textfsm.TextFSM(fh)

parser.ParseText(
""" address-family ipv4 vrf EASY-GOING
  no synchronization
  neighbor 10.42.80.2 remote-as 64442
  neighbor 10.42.80.2 inherit peer-session LTE-ASR
  neighbor 10.42.80.2 description *** some EASY-GOING router ***
  neighbor 10.42.80.2 activate
  neighbor 10.42.80.2 inherit peer-policy LTE-ASR
  neighbor 10.42.80.2 default-originate
 exit-address-family
 !
 address-family ipv4 vrf THE-MONTAIN
  import path selection all
  import path limit 8
  no synchronization
  redistribute static route-map advertise-primary
  redistribute connected route-map advertise-primary
  neighbor 10.144.139.165 remote-as 65021
  neighbor 10.144.139.165 description *** peering CASE DMZ MONTAINVPDMZ01 ***
  neighbor 10.144.139.165 route-map RESILIENCE_VIPS_EXT
  neighbor 10.144.139.166 remote-as 65021
  neighbor 10.144.139.166 ebgp-multihop 5
  neighbor 10.144.139.166 update-source Vlan508
  neighbor 10.144.139.166 route-map DENY_ALL out
 exit-address-family""")

This results in the following:

[['EASY-GOING', '10.42.80.2', '64442', '*** some EASY-GOING router ***', '', '', ''],
 ['THE-MONTAIN', '10.144.139.165', '65021', '*** peering CASE DMZ MONTAINVPDMZ01 ***', 'RESILIENCE_VIPS_EXT', '', ''],
 ['THE-MONTAIN', '10.144.139.166', '65021', '', 'DENY_ALL', 'Vlan508', '5']]

Changes to the FSM template:

Value Required NEIGHBOR (\S+)    # Make required so we don't end up with an empty row at the end due to exit-address-family triggering a record 
Value DESCRIPTION (.*)    # Wasn't matching the example descriptions you provided in config.

 ^ address-family ipv4 vrf ${VRF} -> NEIGHBORSTART    # Jump to the new state.

NEIGHBORSTART     # New state to ensure that the new neighbor attributes are scanned on the first pass.
  ^  neighbor ${NEIGHBOR} remote-as ${REMOTE_AS} -> NEIGHBOR     # New state to ensure that the new neighbor attributes are scanned on the first pass.

NEIGHBOR
  ^  neighbor \S+ remote-as \d+ -> Continue.Record    # When a new `remote-as` keyword encountered, a new neighbor record starts, but we continue to capture the new value after the current record (previous neighbor) is captured.
  ^  neighbor ${NEIGHBOR} remote-as ${REMOTE_AS}    # Moved to the bottom to make sure the first pass catches all of the first neighbor attributes. May not be necessary, but needs to at least be under the previous line.
Geoffrey42 commented 5 years ago

Thanks for the reply @jrschneider !

Your template works fine with my example indeed but I realize while reading your answer that I put a remote-as in both addess-family. Actually in the real data I am dealing with there are some address-families without remote-as. There is absolutely no pattern I can hang on to.

I hope you found an answer to your issue. I really think there should have some Value Option to record only if a Value's value has changed.

harro commented 5 years ago

Would this be a reference input for this problem? Where case 'B' has a 'remote-as' that is not the initial entry, and case 'C' is an instance with no remote-as.

A A1 a1 A1 A2 A2 a2 A2 B B1 B1 b1 C C1 C1

With a desired output of: A, A1, a1 A, A2, a2 B, B1, b1 C, C1

gachteme commented 5 years ago

I can see this as a useful feature to reduce template complexity in this situation and would allow a sensible output that the engine couldn't produce in certain other cases which would likely not occur in networking. Of course you could always have it record like crazy and wouldn't be too hard to parse it out in post (if I recall using lists and a post-processor was my solution to some of these problems).

Might it be useful to treat the feature request and help request separately?

For example, if we have something like you described where a/b is an IP, 1/2/3 is a non-recorded keyword, and a1 is "64442" , etc.

a, 1, a1
a, 2, a2
a, 3, a3
b, 1, b1
b, 2, b2
b, 3, b3

now consider:

a, 1, a1
a, 2, a2
b, 3, b3

How would we distinguish between A2 -> A3 versus A2 -> B3 when A3 and B3 are the same pattern: <IP> route-map <TEXT>?

I've rarely found this in actual router outputs because if B3 existed I've always seen either A3 or B1/B2 exist as well but there have been instances where numerous additional states were required to differentiate between A and B, where this feature would handle it in one state.

diepes commented 3 years ago

i found this thread searching for similar sollution for https://github.com/networktocode/ntc-templates/issues/784

as i understand the issue, there is no stable match for final line to reliably "-> record" and start new record

What is requires is a new action or Value e.g. Action=newOnChange , or Value=newTrigger)

Value newTrigger SEQ (\S+)
Value Filldown A (\S+)
Value Filldown B (\S+)
Value Filldown C (\S+)
Value Filldown D (\S+)
Value Filldown E (\S+)
Value Filldown F (\S+)

Start
  ^${SEQ} info A ${A} -> newOnChange ${SEQ}
  ^${SEQ} info B ${B} -> newOnChange ${SEQ}
  ^${SEQ} info C ${C} -> newOnChange ${SEQ}
   ...
  ^${SEQ} info F ${F} -> newOnChange ${SEQ}

Another example to try and explain the issue.

Example pseudo data, only trigger for new "-> record" is change in 1st value 10 info A 10 info B 10 info C 20 info A 30 info C 40 info C 40 info E 40 info F

output 10, A=A, B=B, C=C, E="", F="" 20, A=A, B="", C="", E="", F="" 30, A="", B="", C=C, E="", F="" 40, A="", B="", C=C, E="", F=F