get_mac_address_table issue processing multicast entries header

[sjtarik]

Description of Issue/Question

When called, get_mac_address_table throws exception while processing the multicast entries header from "show mac address-table" output

Did you follow the steps from https://github.com/napalm-automation/napalm#faq

• [x] Yes
• [ ] No

Setup

napalm-ios version

(Paste verbatim output from pip freeze | grep napalm-ios between quotes below)

(devs) tarik-mbp:dev tarik$ pip freeze | grep napalm-ios
napalm-ios==0.4.0

Since 0.4.0 only issues "show mac-address-table" cli, I install the library directly from github with pip install git+git://github.com/napalm-automation/napalm-ios.git

IOS version

(Paste verbatim output from show version between quotes below)

Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500es8-UNIVERSALK9-M), Version 03.03.02.XO RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Tue 19-Aug-14 02:17 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2013 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0.  The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.  For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: 15.1(1r)SG2

Steps to Reproduce the Issue

Error Traceback

(Paste the complete traceback of the exception between quotes below)

Traceback (most recent call last):
  File "devs/lib/python2.7/site-packages/napalm_ios/ios.py", line 1429, in get_mac_address_table
    raise ValueError("Unexpected output from: {}".format(repr(line)))
ValueError: Unexpected output from: u'Multicast Entries'

[sjtarik]

360 bbc2.deab.1717 dynamic ip,ipx,assigned,other Port-channel1

Multicast Entries vlan mac address type ports ---------+---------------+-------+-------------------------------------------- 1 0100.0ccc.ccce system Po1 1 ffff.ffff.ffff system Po1 39 ffff.ffff.ffff system Gi10/31,Gi10/32,Switch,Po1

[sjtarik]

I realized that there are three version of the same command "show mac address-table" for 3k, 4k and 6k. Copy pasting here:

vlan mac address type learn age ports vlan mac address type protocols port Vlan Mac Address Type Ports

Same thing with the "show lldp neighbors" command. Depending on how they handle word wrapping, some platforms just use a single line for host name and then all the rest goes to a second line. Some other platforms just use all the space between current column and next one until runs out of space. Hard to handle in generic way.

[ktbyers]

@sjtarik Can you re-test this using napalm-ios 0.5.0? I did a lot of work recently on this.

For any that fail, please post the show command and its corresponding output.

[sjtarik]

@ktbyers Kirk, I think the number of fields are right but for exception cases still getting exceptions in all three cases.

u'Total Mac Addresses for this criterion: 320' u'* --- 0000.0000.0000 static No - Router' u'Multicast Entries'

I will try to append cli outputs and exception cases when I have more time. I think we can follow the textfsm approach, if don't know how to handle the line, just silently skip, don't throw exceptions?

[ktbyers]

@sjtarik Yes, please post the output from the commands (and the commands you used to generate the output). It should be pretty easy to fix it from that.

if don't know how to handle the line, just silently skip, don't throw exceptions

Yes, I have mixed emotions about that...that is a good way for your parsing to be broke and not know about it (i.e. you are getting errors and you are just ignoring them).

[dbarrosop]

Yes, I have mixed emotions about that...that is a good way for your parsing to be broke and not know about it (i.e. you are getting errors and you are just ignoring them).

I agree with the statement. I'd rather fail than having wrong/missing data unknowingly. If the issue was too occurrent we could evaluate having a force mode or something as an optional_arg to ignore these problems rather than throwing an Exception but AFAICT this hasn't been a huge issue, has it?

[sjtarik]

Personally I always like degraded but still available service :) rather than an exception but I think in a professional automation environment throwing an exception makes more sense. But my point is there will be many other exceptions depending on the features enabled and the topology. I think we can point out in documentation that we are really strict on parsing/processing in driver functions but workaround is to call the free form cli function and get the exact output, so user can implement the parsing logic she prefers herself.

[dbarrosop]

Problem is that if you relay on automation and data to take informed decisions, you expect that data to be available and correct or, at least, know when it might not be so a human can intervene. I think the right thing to do is detect the issue by failing and fix it ASAP. If it can't be fixed, then treat it somehow and have this "degraded service" as you say.

[ktbyers]

We also need the feedback loop in NAPALM i.e. we need to get feedback that it is not working properly so the we can improve the getters.

[ktbyers]

@sjtarik I still need the "show mac address-table" output from the failing devices....I need this so I can fix the problem. I also need it so I can incorporate that output (and possibly different show commands into the unit tests).

[sjtarik]

@ktbyers Kirk, sorry I am still traveling. I will update the thread on 29th. Happy holidays.

[ktbyers]

@sjtarik No problem :-)

[sjtarik]

@ktbyers Attaching the sample data I collected. Happy new year. mac_sample_1.txt mac_sample_2.txt mac_sample_3.txt

[itdependsnetworks]

@ktbyers I know you are against textfsm, but here is a there solution we came up with from same issue.

https://github.com/networktocode/ntc-templates/blob/master/templates/cisco_ios_show_mac-address-table.template

[ktbyers]

@itdependsnetworks Yes, issue is not really fixing it...that is easy enough. Issue is getting the sample of the possible output formats for a given command.

That being said, I haven't fixed this yet :-)

[ktbyers]

@sjtarik You stated above you were getting exceptions due to the following three lines (in different command outputs)...unless I am misreading what you put above.

u'Total Mac Addresses for this criterion: 320'
u'* --- 0000.0000.0000 static No - Router'
u'Multicast Entries'

Yet, the three output files you provided had none of the above lines in them?

I ran NAPALM through the three outputs you provided and only saw one minor issue (mac_sample_1.txt would set the interface to 'Router' for a static field). None of the above outputs caused a NAPALM exception.

Can you provide the actual failing output i.e. the 'show mac address table' command output that causes NAPALM to fail?

[ktbyers]

I did reverse engineer and fix this one output format (but I would rather have real output from a device):

Unicast Entries
 vlan     mac address     type        protocols               port
---------+---------------+--------+---------------------+-------------------------
   1      30a3.30a3.a1c3   dynamic ip,ipx,assigned,other Port-channel1
  99      30a3.30a3.a1c4   dynamic ip,ipx,assigned,other Port-channel1
  99      30a3.30a3.a1c5   dynamic ip,ipx,assigned,other Port-channel1
  99      30a3.30a3.a1c6   dynamic ip,ipx,assigned,other Port-channel1
  99      30a3.30a3.a1c7   dynamic ip,ipx,assigned,other Port-channel1
  99      30a3.30a3.a1c8   dynamic ip,ipx,assigned,other Port-channel1
  99      30a3.30a3.a1c9   dynamic ip,ipx,assigned,other Port-channel1
  99      30a3.30a3.a1ca   dynamic ip,ipx,assigned,other Port-channel1

Multicast Entries
vlan mac address type ports
---------+---------------+-------+--------------------------------------------
1 0100.0ccc.ccce system Po1
1 ffff.ffff.ffff system Po1
39 ffff.ffff.ffff system Gi10/31,Gi10/32,Switch,Po1

[ktbyers]

Okay, I found this second format for a 3560 (which covers one of your cases):

show mac address-table
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
All    0100.0ccc.cccc    STATIC      CPU
All    0100.0ccc.cccd    STATIC      CPU
All    0180.c200.0000    STATIC      CPU
All    0180.c200.0001    STATIC      CPU
All    0180.c200.0002    STATIC      CPU
All    0180.c200.0003    STATIC      CPU
All    0180.c200.0004    STATIC      CPU
All    0180.c200.0005    STATIC      CPU
All    0180.c200.0006    STATIC      CPU
All    0180.c200.0007    STATIC      CPU
All    0180.c200.0008    STATIC      CPU
All    0180.c200.0009    STATIC      CPU
All    0180.c200.000a    STATIC      CPU
All    0180.c200.000b    STATIC      CPU
All    0180.c200.000c    STATIC      CPU
All    0180.c200.000d    STATIC      CPU
All    0180.c200.000e    STATIC      CPU
All    0180.c200.000f    STATIC      CPU
All    0180.c200.0010    STATIC      CPU
All    ffff.ffff.ffff    STATIC      CPU
   1    000a.b82d.10e0    DYNAMIC     Fa0/16
   1    0012.80b6.4cd8    DYNAMIC     Fa0/3
   1    0012.80b6.4cd9    DYNAMIC     Fa0/16
   1    0014.6915.4100    DYNAMIC     Fa0/16
   1    0018.b921.9200    DYNAMIC     Fa0/16
   1    0018.b921.9278    DYNAMIC     Fa0/1
   1    0018.b974.528f    DYNAMIC     Fa0/16
   1    0019.0617.660f    DYNAMIC     Fa0/13
   1    0019.0617.6610    DYNAMIC     Fa0/14
   1    0019.0617.6611    DYNAMIC     Fa0/15
   1    001b.d450.970f    DYNAMIC     Fa0/19
   1    001b.d450.9710    DYNAMIC     Fa0/20
   1    001b.d450.9711    DYNAMIC     Fa0/21
   4    0018.b974.528f    DYNAMIC     Fa0/16
  45    0018.b945.d5a9    DYNAMIC     Fa0/19
  45    0018.b945.f780    DYNAMIC     Fa0/5
  45    0018.b974.528f    DYNAMIC     Fa0/16
  56    0018.b945.f781    DYNAMIC     Fa0/16
  56    0018.b974.528f    DYNAMIC     Fa0/16
  56    0019.069c.80e1    DYNAMIC     Fa0/19
   6    0018.b974.528f    DYNAMIC     Fa0/16
   6    0019.069c.80e0    DYNAMIC     Fa0/13
Total Mac Addresses for this criterion: 42

[ktbyers]

@sjtarik So the one I am missing now is this one:

u'* --- 0000.0000.0000 static No - Router'

Which platform is this from and do you have the full output?

[ktbyers]

@sjtarik See this PR https://github.com/napalm-automation/napalm-ios/pull/95

Can you test the new code and see if it fixes the issues you were running into? The new code is here:

https://github.com/ktbyers/napalm-ios/tree/devel

[sjtarik]

@ktbyers Hi Kirk, sorry pretty busy week. I will take care of it this weekend.

[ktbyers]

@sjtarik No problem. The fix is in the napalm-ios develop branch now.

https://github.com/napalm-automation/napalm-ios/

[ktbyers]

I think this is all fixed.