search

networktocode / ntc-templates

TextFSM templates for parsing show commands of network devices
https://ntc-templates.readthedocs.io/
Other
1.11k stars 729 forks source link

cisco_ios_show_interfaces_status.textfsm parse error for PortChannels #1606

Closed kuraijay closed 4 months ago

kuraijay commented 10 months ago
ISSUE TYPE
TEMPLATE USING
Value PORT (\S+)
Value NAME (.+?)
Value STATUS (err-disabled|disabled|connected|notconnect|inactive|up|down|monitoring|suspended)
Value VLAN_ID (\d+|trunk|routed|unassigned)
Value DUPLEX (\S+)
Value SPEED (\S+)
Value TYPE (.*)
Value FC_MODE (\S+)

Start
  ^Load\s+for\s+
  # Capture time-stamp if vty line has command time-stamping turned on
  ^Time\s+source\s+is
  ^-+\s*$$
  ^Port\s+Name\s+Status\s+Vlan\s+Duplex\s+Speed\s+Type -> Interfaces
  ^\s*$$
  ^. -> Error

Interfaces
  #Match fc...
  ^\s*${PORT}\s+is\s+${STATUS}\s+Port\s+mode\s+is\s+${FC_MODE}\s*$$ -> Record
  ^\s*${PORT}\s+is\s+${STATUS}\s+\(${TYPE}\)\s*$$ -> Record
  ^\s*${PORT}\s+${STATUS}\s+${VLAN_ID}\s+${DUPLEX}\s+${SPEED}\s*${TYPE}$$ -> Record
  ^\s*${PORT}\s+${NAME}\s+${STATUS}:\s+${VLAN_ID}\s+${DUPLEX}\s+${SPEED}\s*${TYPE}$$ -> Record
  ^\s*${PORT}\s+${NAME}\s+${STATUS}\s+${VLAN_ID}\s+${DUPLEX}\s+${SPEED}\s*${TYPE}$$ -> Record
  ^-+
  ^\s*$$
  ^. -> Error
SAMPLE COMMAND OUTPUT
Port      Name               Status       Vlan       Duplex  Speed Type 

Gi4/0/9   disabled     1            auto   auto 10/100/1000BaseTX SFP 

Gi4/0/10  wireless disabled     1            auto   auto 10/100/1000BaseTX SFP 

Gi4/0/11  wireless disabled     1            auto   auto 10/100/1000BaseTX SFP 

Gi4/0/12  wireless disabled     1            auto   auto 10/100/1000BaseTX SFP 

Po3       Wireless Controlle notconnect   unassigned   auto   auto
SUMMARY

Script runs the show interfaces status command and text parses the data. anything that has the Po (port channel) interfaces throws the error

textfsm.parser.TextFSMError: State Error raised. Rule Line: 28. Input Line: Po3 Wireless Controlle notconnect unassigned auto auto

STEPS TO REPRODUCE
Write a script that will run the command "show interfaces status" then use TextFSM to parse it.
EXPECTED RESULTS
{
    "DUPLEX": "auto",
    "FC_MODE": "",
    "NAME": "Wireless Controller",
    "PORT": "Po3",
    "SPEED": "auto",
    "STATUS": "notconnect",
    "TYPE": "",
    "VLAN_ID": "unassigned"
}
ACTUAL RESULTS
textfsm.parser.TextFSMError: State Error raised. Rule Line: 28. Input Line: Po3       Wireless Controlle notconnect   unassigned   auto   auto
kuraijay commented 10 months ago

Figured it out looks like the template is looking for TYPE when its blank for the "Po" interfaces.

I added these two lines and tested and it works:

^\s${PORT}\s+${NAME}\s+${STATUS}\s+${VLAN_ID}\s+${DUPLEX}\s+${SPEED}\s$$ -> Record ^\s${PORT}\s+${NAME}\s+${STATUS}\s+${VLAN_ID}\s+${DUPLEX}\s$$ -> Record

Maybe we can modify the repo's template to skip the type?

mjbear commented 9 months ago

Maybe we can modify the repo's template to skip the type?

@kuraijay I did some quick testing via https://textfsm.nornir.tech/ with your example data and the template from above (the lines include the TYPE capture group) and it parses fine.

I also proceeded to confirm it parses via the NTC yaml generator script (cli.py) and it does as well.

Might you be able to share another block of sample command output and the error that is thrown by textfsm?

kuraijay commented 9 months ago

Maybe we can modify the repo's template to skip the type?

@kuraijay I did some quick testing via https://textfsm.nornir.tech/ with your example data and the template from above (the lines include the TYPE capture group) and it parses fine.

I also proceeded to confirm it parses via the NTC yaml generator script (cli.py) and it does as well.

Might you be able to share another block of sample command output and the error that is thrown by textfsm?

it's the only sample I got sorry. It parses fine on the website, but when I went to write the script to run the command and use the template that's the only time it errored. Maybe there some kind of value it sees when disappears when you copy it over.

mjbear commented 9 months ago

Hello @kuraijay

I'd like to help figure this out so please bear with me while I ask some questions below. Thank you! :smiley:

Together we'll solve this. Thank you!

mjbear commented 9 months ago

Hello @kuraijay

I've done some digging and located output where the type for port-channels is N/A instead of no text or white space as your output showed.

:arrow_right: Answers to questions in my comment from a few days ago will be extremely helpful in order for me to help solve the templating issue.

kuraijay commented 9 months ago
  • What Cisco IOS version?
  • What Cisco hardware model?
  • Edit: What version of NTC templates is being used?
  • Did you have to make any modifications to the output to remove confidential text or was that a direct copy+paste?

  • What are the modules and versions of those modules is your script using?

    • If you're using a Python virtual environment, please provide the output from pip freeze.

  • Would you be willing to save the raw output to a plain text file and attach it on this thread?

    • (There's the paper clip icon to attach a file on this Github discussion.)

What Cisco IOS version? 16.3.6 What Cisco hardware model? C3850 Edit: What version of NTC templates is being used? 4.1.0 Did you have to make any modifications to the output to remove confidential text or was that a direct copy+paste? No

What are the modules and versions of those modules is your script using? If you're using a Python virtual environment, please provide the output from pip freeze.

asttokens==2.2.1 
backcall==0.2.0 
bcrypt==4.0.1 
cffi==1.16.0 
colorama==0.4.6 
comm==0.1.2 
cryptography==41.0.5 
debugpy==1.6.5 
decorator==5.1.1 
entrypoints==0.4 
et-xmlfile==1.1.0 
executing==1.2.0 
future==0.18.3 
ipykernel==6.20.1 
ipython==8.8.0 
jedi==0.18.2 
jupyter_client==7.4.9 
jupyter_core==5.1.3 
matplotlib-inline==0.1.6 
nest-asyncio==1.5.6 
netmiko==4.3.0 
ntc_templates==4.1.0 
numpy==1.26.3 
openpyxl==3.1.2 
packaging==23.0 
pandas==2.2.0 
paramiko==3.3.1 
parso==0.8.3 
pickleshare==0.7.5 
platformdirs==2.6.2 
prompt-toolkit==3.0.36 
psutil==5.9.4 
pure-eval==0.2.2 
pyarrow==15.0.0 
pycparser==2.21 
Pygments==2.14.0 
PyNaCl==1.5.0 
pyOpenSSL==23.3.0 
pyserial==3.5 
python-dateutil==2.8.2 
pytz==2023.3.post1 
pywin32==303 
PyYAML==6.0.1 
pyzmq==25.0.0 
scp==0.14.5 
six==1.16.0 
stack-data==0.6.2 
textfsm==1.1.3 
tornado==6.2 
traitlets==5.8.1
tzdata==2023.4 
wcwidth==0.2.5 
XlsxWriter==3.1.9

Would you be willing to save the raw output to a plain text file and attach it on this thread? (There's the paper clip icon to attach a file on this Github discussion.) C3850_Output.txt

mjbear commented 9 months ago

@kuraijay Thank you!

I took your plaintext file, removed the host name prompts, the --more--, and the extra column headers. Then I ran that entire long file through NTC's cli.py and it generates the structured yaml output with no issues.

I'm not able to identify a discrepancy in the output.

Based on the output from git blame, 7 years ago the TYPE was changed to the regex it is now to work with port channels.

  - duplex: "auto"
    fc_mode: ""
    name: "member of wireless"
    port: "Gi4/0/11"
    speed: "auto"
    status: "disabled"
    type: "10/100/1000BaseTX SFP"
    vlan_id: "1"
  - duplex: "auto"
    fc_mode: ""
    name: "member of wireless"
    port: "Gi4/0/12"
    speed: "auto"
    status: "disabled"
    type: "10/100/1000BaseTX SFP"
    vlan_id: "1"
  - duplex: "auto"
    fc_mode: ""
    name: "Wireless Controlle"
    port: "Po3"
    speed: "auto"
    status: "notconnect"
    type: ""
    vlan_id: "unassigned"
mjbear commented 8 months ago

@kuraijay The good news is I found a decommissioned Catalyst 3650 with 16.3.6 code still on it. It's not a 3850, but it's the same family and the same code revision.

The unfortunate news is that the output I retrieved from it parses fine just like earlier raw output examples.

Your Jan 24 comment mentions the TYPE capture group. The regex for TYPE is .* and with that being zero or more of any character. There could be nothing in the Type column and the regex will still function.

I can't find an issue with the template. :disappointed:

:arrow_right: Based on your pip freeze output I figure you're using netmiko's use_textfsm argument to call TextFSM?

Please double check that your script is not accidentally referencing an old version of ntc-templates. You might go as far as manually referencing a git clone copy of ntc-templates during testing to rule things out.

Hope this helps.

kuraijay commented 8 months ago

@kuraijay The good news is I found a decommissioned Catalyst 3650 with 16.3.6 code still on it. It's not a 3850, but it's the same family and the same code revision.

The unfortunate news is that the output I retrieved from it parses fine just like earlier raw output examples.

Your Jan 24 comment mentions the TYPE capture group. The regex for TYPE is .* and with that being zero or more of any character. There could be nothing in the Type column and the regex will still function.

I can't find an issue with the template. 😞

➡️ Based on your pip freeze output I figure you're using netmiko's use_textfsm argument to call TextFSM?

Please double check that your script is not accidentally referencing an old version of ntc-templates. You might go as far as manually referencing a git clone copy of ntc-templates during testing to rule things out.

Hope this helps.

Was this done still copying and pasting the output? or did you have a script to run the command and usetextfsm?

mjbear commented 8 months ago

Was this done still copying and pasting the output? or did you have a script to run the command and usetextfsm?

In this specific case, I captured the text in a file since it was no longer network-connected. I'll take this further soon to plumb the 16.3.6 switch up temporarily so I can validate with the device directly.

mjbear commented 8 months ago

@kuraijay

A day ago I fired up a 3650 running 6.3.6 code and added a port-channel interface. Next I ran a quick netmiko script (using the show interfaces status command) against the switch and it parsed into structured text successfully. Ironically the Python virtual environment I used had an older version of netmiko and ntc-templates. I updated netmiko and ntc-templates to the latest available versions via pip to find it still parsed successfully.

If you haven't yet,

  1. Please try running your script against another 3850 switch running the same 16.3.6 code release.
  2. Please create a new Python virtual environment from scratch then install ntc-templates and netmiko. Activate this new minimal virtual environment and run your script. See what the outcome is.
  3. If we suspect a text editor is converting characters during a copy+paste, maybe set up your script to write the raw output (from netmiko) to file. This way you can review the raw output in Notepad++, vim, or other editors which support viewing special characters. What do you think?

Hope this helps!

mjbear commented 5 months ago

Hello @kuraijay ,

Circling back on this. Were you able to run through any of the suggestions?

Thank you!

mjbear commented 4 months ago

@kuraijay Did you find out anything new regarding the show interfaces status parsing issue you saw?

Should we resolve this issue ticket?

Thank you for responding!