get_bgp_neighbors method re-implementation

[benmaddison]

The revised implementation provides:

• ipv6 address family support
• neighbor description parsing
• order of magnitude speed up in execution time for routers with many peers

textfsm is used for output parsing. I'm willing to help re-write using re if the consensus is against the use of textfsm.

[coveralls]

Coverage decreased (-0.7%) to 66.479% when pulling b29e68356a668b3436ce9ab9091949490c70b312 on wolcomm:develop into 1069480b7b1054d2070fba8b32b11a01c4fa333c on napalm-automation:develop.

[coveralls]

Coverage decreased (-0.7%) to 66.479% when pulling b29e68356a668b3436ce9ab9091949490c70b312 on wolcomm:develop into 1069480b7b1054d2070fba8b32b11a01c4fa333c on napalm-automation:develop.

[coveralls]

Coverage decreased (-0.7%) to 66.479% when pulling b29e68356a668b3436ce9ab9091949490c70b312 on wolcomm:develop into 1069480b7b1054d2070fba8b32b11a01c4fa333c on napalm-automation:develop.

[coveralls]

Coverage increased (+1.8%) to 69.007% when pulling 5d88567ccc3a747fdd9c79b8a76ea81e552afef9 on wolcomm:develop into 1069480b7b1054d2070fba8b32b11a01c4fa333c on napalm-automation:develop.

[ktbyers]

I will see if I can re-write this to not use TextFSM. But implement the feature requested and integrate the unit tests.

I don't think it makes sense to add TextFSM especially given the probably long term direction of YANG for the getters.

[coveralls]

Coverage increased (+1.8%) to 69.036% when pulling ceb98eb3ba9ff1066cf084a5d2c107e2d0655679 on wolcomm:develop into 1069480b7b1054d2070fba8b32b11a01c4fa333c on napalm-automation:develop.

[benmaddison]

I'd be surprised if we ever saw a usable yang implementation in ios classic/xe land to be honest. We are going to be scraping screens for a while! Re-implementing using re only shouldn't be that hard, and I'm happy to assist. My strong preference, however, would be to pull this change and then rework in a new PR, since the performance and lack of multi-AF support makes the current version unusable in all but the most trivial networks.

[ktbyers]

YANG is just a modeling language. So NAPALM-Yang will parse configuration/output and convert to YANG abstractions. Or go from YANG models to vendor configs.

The network device doesn't have to be capable of supporting YANG.

So yes, screen-scraping, but to/from YANG models.

[coveralls]

Coverage increased (+2.6%) to 69.827% when pulling 4627d688e24cd57a81460268482ba6af2c7e0527 on wolcomm:develop into 1069480b7b1054d2070fba8b32b11a01c4fa333c on napalm-automation:develop.

[benmaddison]

Anything I can assist with to get this merged? I need to publish a release of a package that has napalm as a dependency in the next few days, but without this fix the performance becomes unacceptably poor for IOS targets.

[ktbyers]

Yes, sorry about delay on this. Let me see if I can review this on Thursday or Friday.

[benmaddison]

Thanks @ktbyers. Let me know if you come across anything you need me to tidy up.

[ktbyers]

Some of the items I mention or mostly from a perspective of keeping things consistent versus one being better than the other (for example, netaddr vs ipaddress, or using conditionals and exceptions instead of assert statements).

[benmaddison]

all comments addressed except the switch from ipaddress to netaddr. will try and get to that later today. let me know if there is anything else you pick up the meantime?

[coveralls]

Coverage increased (+2.9%) to 70.153% when pulling 9490319c0446cf4a9c9d708591259fb6237bfdbf on wolcomm:develop into 1069480b7b1054d2070fba8b32b11a01c4fa333c on napalm-automation:develop.

[benmaddison]

I think that's everything ;-)

[ktbyers]

@benmaddison I am going to merge this change into a separate branch as there are some things that I want to change.

Given the size of the change it is to hard to manage just via feedback on this channel.

I will post the link to the branch after I create it.

[ktbyers]

Here is the branch:

https://github.com/napalm-automation/napalm-ios/tree/get_bgp_neighbors_pr

[ktbyers]

@benmaddison On the BGP summary processing isn't there an issue with the fill-down fields?

It looks like you are propagating the 'uptime', 'remote_as', and 'remote_addr' as fill-down from the BGP summary?

[benmaddison]

@ktbyers those fields are guaranteed to be present if a line in the output is matched against a pattern that has record: True, so they'll never fall through from an earlier match. It won't change the behaviour if we add those to no_fill_fields, so we can do that it you think it makes the intent clearer?

[benmaddison]

On the asdot syntax question, i'm not inclined to spend time supporting something that was deprecated almost 10 years ago (rfc5396), and hasn't been the default in XE since 2.4! Is there need for it that doesn't have an easy workaround that you're aware of?

[benmaddison]

@ktbyers I have no issues with your other clean-ups

[ktbyers]

@benmaddison On the asdot syntax...it is a configuration option, so we are pretty much guaranteed to see AS numbers in this format.

It is a simple fix so I am going to put it in.

You don't have to do it though...I can add that in.

[ktbyers]

@benmaddison Okay, I will probably update the non-fill fields to include others.

Basically, I think this will make the risk lower to propagate values obtained earlier from a different peer (in case the regular expression patterns change, or some other change happens).

[ktbyers]

@benmaddison Why did you choose?

                 # merge other values in a sane manner 
                 existing['is_up'] = existing['is_up'] or is_up
                 existing['uptime'] = max(existing['uptime'], uptime)

I am inclined to make the is_up be a logical and (i.e. if either address families are down, then flag it as down).

Similarly, I am inclined to make the uptime the min of the uptime (i.e. I want to know one of them bounced recently).

Thoughts?

[ktbyers]

New PR here:

https://github.com/napalm-automation/napalm-ios/pull/153