IOS configs sometimes incomplete

[rvandepu]

I've already posted in issue #1401 , but don't want to hijack OP's thread since he has 2 separate issues and this one was apparently resolved.

Problem statement: Sometimes, oxidized is able to retrieve a full config for my IOS devices, sometimes I only get some 'show ver' information, but no config.

The problem occurs on multiple IOS devices. I'll try to narrow down a specific model.

Examples: Non-successful retrieval:

! ! ! Cisco Internetwork Operating System Software ! IOS (tm) C2950 Software (C2950-I6Q4L2-M), Version 12.1(22)EA1, RELEASE SOFTWARE (fc1) ! Copyright (c) 1986-2004 by cisco Systems, Inc. ! Compiled Mon 12-Jul-04 08:18 by madison ! Image text-base: 0x80010000, data-base: 0x8055C000 ! ! ROM: Bootstrap program is C2950 boot loader ! ! BCACTU01 uptime is 6 days, 6 hours, 42 minutes ! System returned to ROM by power-on ! System restarted at 02:37:37 MET Wed Jul 11 2018 ! System image file is "flash:/c2950-i6q4l2-mz.121-22.EA1.bin" ! ! cisco WS-C2950T-24 (RC32300) processor (revision Q0) with 20873K bytes of memory. ! Processor board ID FOC0847Y26J ! Last reset from system-reset ! Running Enhanced Image ! 24 FastEthernet/IEEE 802.3 interface(s) ! 2 Gigabit Ethernet/IEEE 802.3 interface(s) ! ! 32K bytes of flash-simulated non-volatile configuration memory. ! Base ethernet MAC Address: 00:12:80:C2:A4:40 ! Motherboard assembly number: 73-6114-10 ! Power supply part number: 34-0965-01 ! Motherboard serial number: FOC08472EHJ ! Power supply serial number: DAB0843JLZ3 ! Model revision number: Q0 ! Motherboard revision number: A0 ! Model number: WS-C2950T-24 ! System serial number: FOC0847Y26J ! Configuration register is 0xF !

Successful retrieval:

! BCACTU01#show version ! ! Image: Software: C2950-I6Q4L2-M, 12.1(22)EA1, RELEASE SOFTWARE (fc1) ! Image: Compiled: Mon 12-Jul-04 08:18 by madison ! Image: flash:/c2950-i6q4l2-mz.121-22.EA1.bin ! Chassis type: WS-C2950T-24 ! Memory: main 20873K ! Processor ID: FOC0847Y26J ! CPU: RC32300 ! ! show inventory ! ! ! ! ! No configuration change since last restart ! version 12.1 no service pad service timestamps debug uptime service timestamps log uptime service password-encryption ! hostname BCACTU01 -REST OF CONFIG REDACTED-

Troubleshooting I already disabled the retrieval for the VTP information by commenting out those line in the ios model. This did not solve the issue. Below is an example with VTP retrieval still enabled.

! ! ! VTP: Cisco Internetwork Operating System Software ! VTP: IOS (tm) C2950 Software (C2950-I6Q4L2-M), Version 12.1(22)EA1, RELEASE SOFTWARE (fc1) ! VTP: Copyright (c) 1986-2004 by cisco Systems, Inc. ! VTP: Compiled Mon 12-Jul-04 08:18 by madison ! VTP: Image text-base: 0x80010000, data-base: 0x8055C000 ! VTP: ROM: Bootstrap program is C2950 boot loader ! VTP: BCACTU01 uptime is 1 day, 2 hours, 29 minutes ! VTP: System returned to ROM by power-on ! VTP: System restarted at 02:37:37 MET Wed Jul 11 2018 ! VTP: System image file is "flash:/c2950-i6q4l2-mz.121-22.EA1.bin" ! VTP: cisco WS-C2950T-24 (RC32300) processor (revision Q0) with 20873K bytes of memory. ! VTP: Processor board ID FOC0847Y26J ! VTP: Last reset from system-reset ! VTP: Running Enhanced Image ! VTP: 24 FastEthernet/IEEE 802.3 interface(s) ! VTP: 2 Gigabit Ethernet/IEEE 802.3 interface(s) ! VTP: 32K bytes of flash-simulated non-volatile configuration memory. ! VTP: Base ethernet MAC Address: 00:12:80:C2:A4:40 ! VTP: Motherboard assembly number: 73-6114-10 ! VTP: Power supply part number: 34-0965-01 ! VTP: Motherboard serial number: FOC08472EHJ ! VTP: Power supply serial number: DAB0843JLZ3 ! VTP: Model revision number: Q0 ! VTP: Motherboard revision number: A0 ! VTP: Model number: WS-C2950T-24 ! VTP: System serial number: FOC0847Y26J ! VTP: Configuration register is 0xF ! ! VTP Version : 2 ! Configuration Revision : 0 ! Maximum VLANs supported locally : 250 ! Number of existing VLANs : 11 ! VTP Operating Mode : Transparent ! VTP Domain Name : idtv ! VTP Pruning Mode : Disabled ! VTP V2 Mode : Disabled ! VTP Traps Generation : Disabled ! MD5 digest : 0x40 0x20 0x49 0xD7 0xC6 0x89 0x94 0xF5 ! Configuration last modified by 0.0.0.0 at 0-0-00 00:00:00

In the good configs, oxidized parses the 'show ver' information and displays a summary. In the non-successful examples, the raw 'show ver' output is displayed. With the VTP lines still uncommented, the 'show ver' output seems to sometimes be parsed by the VTP part of the code?

I've also noticed, that in the good configs, the top line varies between:

! BCACTU01#show version and ! Cisco Internetwork Operating System Software

Any idea what could be causing this? Since it's telnet, I'll get some wireshark output..

[ytti]

Neither of examples are success.

In claimed successful example you have prompt and show version visible, these are removed by: https://github.com/ytti/oxidized/blob/0.24.0/lib/oxidized/model/ios.rb#L23

That line removes first (show version) and last line (prompt). And there is no reason to suspect it has not done so. So logical conclusion is, we are confused what is first and what is last line, this happens when we fail to consume prompt correctly.

How does the session start look like, is there any chance there is MOTD or such which might be considered prompt mistakenly?

[rvandepu]

There are some successful examples, I think:

! Cisco Internetwork Operating System Software ! ! Image: Software: C2950-I6Q4L2-M, 12.1(22)EA1, RELEASE SOFTWARE (fc1) ! Image: Compiled: Mon 12-Jul-04 08:18 by madison ! Image: flash:/c2950-i6q4l2-mz.121-22.EA1.bin ! Chassis type: WS-C2950T-24 ! Memory: main 20873K ! Processor ID: FOC0847Y26J ! CPU: RC32300 ! ! show inventory ! ! ! ! ! No configuration change since last restart ! version 12.1 no service pad service timestamps debug uptime service timestamps log uptime service password-encryption ! hostname BCACTU01 -REST OF CONFIG REDACTED-

A manual run of these commands looks like:

Trying -REDACTED-... Connected to BCACTU01-REDACTED-. Escape character is '^]'.

Username:-REDACTED- Password:

BCACTU01#show ver Cisco Internetwork Operating System Software IOS (tm) C2950 Software (C2950-I6Q4L2-M), Version 12.1(22)EA1, RELEASE SOFTWARE (fc1) Copyright (c) 1986-2004 by cisco Systems, Inc. Compiled Mon 12-Jul-04 08:18 by madison Image text-base: 0x80010000, data-base: 0x8055C000

ROM: Bootstrap program is C2950 boot loader

BCACTU01 uptime is 6 days, 8 hours, 4 minutes System returned to ROM by power-on System restarted at 02:37:37 MET Wed Jul 11 2018 System image file is "flash:/c2950-i6q4l2-mz.121-22.EA1.bin"

cisco WS-C2950T-24 (RC32300) processor (revision Q0) with 20873K bytes of memory. Processor board ID FOC0847Y26J Last reset from system-reset Running Enhanced Image 24 FastEthernet/IEEE 802.3 interface(s) 2 Gigabit Ethernet/IEEE 802.3 interface(s)

32K bytes of flash-simulated non-volatile configuration memory. Base ethernet MAC Address: 00:12:80:C2:A4:40 Motherboard assembly number: 73-6114-10 Power supply part number: 34-0965-01 Motherboard serial number: FOC08472EHJ Power supply serial number: DAB0843JLZ3 Model revision number: Q0 Motherboard revision number: A0 Model number: WS-C2950T-24 System serial number: FOC0847Y26J Configuration register is 0xF

BCACTU01#show vtp status VTP Version : 2 Configuration Revision : 0 Maximum VLANs supported locally : 250 Number of existing VLANs : 11 VTP Operating Mode : Transparent VTP Domain Name : idtv VTP Pruning Mode : Disabled VTP V2 Mode : Disabled VTP Traps Generation : Disabled MD5 digest : 0x40 0x20 0x49 0xD7 0xC6 0x89 0x94 0xF5 Configuration last modified by 0.0.0.0 at 0-0-00 00:00:00 BCACTU01#show inventory ^ % Invalid input detected at '^' marker.

BCACTU01#show run BCACTU01#show running-config Building configuration...

Current configuration : 5373 bytes ! ! No configuration change since last restart ! version 12.1 no service pad service timestamps debug uptime service timestamps log uptime service password-encryption ! hostname BCACTU01 -REDACTED-

[ytti]

The show inventory shouldn't appear in successful run. I think we are having prompt detection issue here.

Anywhere in raw input is there > or # character outside prompt, which we might read as prompt?

[rvandepu]

Alright, I've just captured the telnet session and debug log for what seems like a good and bad run. Bad run:

........................ Username:...-REDACTED- ...-REDACTED- Password:...-REDACTED-

BCACTU01#enable e .nable BCACTU01#terminal length 0 . BCACTU01#tterminal width 0 .erminal length 0 BCACTU01#show version .terminal width 0 BCACTU01#show inventory .show version Cisco Internetwork Operating System Software IOS (tm) C2950 Software (C2950-I6Q4L2-M), Version 12.1(22)EA1, RELEASE SOFTWARE (fc1) Copyright (c) 1986-2004 by cisco Systems, Inc. Compiled Mon 12-Jul-04 08:18 by madison Image text-base: 0x80010000, data-base: 0x8055C000

ROM: Bootstrap program is C2950 boot loader

BCACTU01 uptime is 6 days, 8 hours, 17 minutes System returned to ROM by power-on System restarted at 02:37:37 MET Wed Jul 11 2018 System image file is "flash:/c2950-i6q4l2-mz.121-22.EA1.bin"

cisco WS-C2950T-24 (RC32300) processor (revision Q0) with 20873K bytes of memory. Processor board ID FOC0847Y26J Last reset from system-reset Running Enhanced Image 24 FastEthernet/IEEE 802.3 interface(s) 2 Gigabit Ethernet/IEEE 802.3 interface(s)

32K bytes of flash-simulated non-volatile configuration memory. Base ethernet MAC Address: 00:12:80:C2:A4:40 Motherboard assembly number: 73-6114-10 Power supply part number: 34-0965-01 Motherboard serial number: FOC08472EHJ Power supply serial number: DAB0843JLZ3 Model revision number: Q0 Motherboard revision number: A0 Model number: WS-C2950T-24 System serial number: FOC0847Y26J Configuration register is 0xF

BCACTU01#show running-config .show inventory show inventory ^ % Invalid input detected at '^' marker.

BCACTU01#exit show runni

Good run:

........................ Username:...-REDACTED- ...-REDACTED- Password:-REDACTED-

BCACTU01#enable e .nable BCACTU01# terminal length 0 .BCACTU01#terminal length 0 BCACTU01#terminal width 0 .terminal width 0 BCACTU01#show version .show version Cisco Internetwork Operating System Software IOS (tm) C2950 Software (C2950-I6Q4L2-M), Version 12.1(22)EA1, RELEASE SOFTWARE (fc1) Copyright (c) 1986-2004 by cisco Systems, Inc. Compiled Mon 12-Jul-04 08:18 by madison Image text-base: 0x80010000, data-base: 0x8055C000

ROM: Bootstrap program is C2950 boot loader

BCACTU01 uptime is 6 days, 8 hours, 16 minutes System returned to ROM by power-on System restarted at 02:37:37 MET Wed Jul 11 2018 System image file is "flash:/c2950-i6q4l2-mz.121-22.EA1.bin"

cisco WS-C2950T-24 (RC32300) processor (revision Q0) with 20873K bytes of memory. Processor board ID FOC0847Y26J Last reset from system-reset Running Enhanced Image 24 FastEthernet/IEEE 802.3 interface(s) 2 Gigabit Ethernet/IEEE 802.3 interface(s)

32K bytes of flash-simulated non-volatile configuration memory. Base ethernet MAC Address: 00:12:80:C2:A4:40 Motherboard assembly number: 73-6114-10 Power supply part number: 34-0965-01 Motherboard serial number: FOC08472EHJ Power supply serial number: DAB0843JLZ3 Model revision number: Q0 Motherboard revision number: A0 Model number: WS-C2950T-24 System serial number: FOC0847Y26J Configuration register is 0xF

BCACTU01#show inventory .show inventory show inventory ^ % Invalid input detected at '^' marker.

BCACTU01#show running-config .show running-config Building configuration...

Current configuration : 5373 bytes ! ! No configuration change since last restart ! version 12.1 no service pad service timestamps debug uptime service timestamps log uptime service password-encryption ! hostname BCACTU01 -REDACTED- end

BCACTU01#exit exit

In the 'bad' example, it seems like the 'show inventory' command is executed before the output of 'show version' is returned.

If we dig deeper in the telnet session, you can see that the Cisco returns a prompt before echoing the 'terminal length 0' back to the server. It does this again with terminal width. If you look at the 'enable' command, you can see that it only echoes back the 'enable'-letters and then an empty prompt. I'll get a pcap from another Cisco model to see if this is model-specific...

I can get you the raw pcaps, if you like, it's quite more visible that way.

I don't see anything that could be mistaken for the prompt in the raw input.

[ytti]

Why do you have enable on, when you are already enabled? I think this is where it breaks.

We could make the IOS model more robust against this type of, what I view misconfig and change: https://github.com/ytti/oxidized/blob/0.24.0/lib/oxidized/model/ios.rb#L135-L139 to something like https://github.com/ytti/oxidized/blob/master/lib/oxidized/model/netgear.rb#L18-L23

With of course enable prompt changed to look what it looks like in IOS.

[ytti]

The crucial problem here being, the model does send "enable\n", which means we blindly send enable, we don't expect anything. But implicitly we expect to see enable prompt. The next line is cmd vars(:enable) which sends your enable password, and expects to see prompt.

In your case enable already returns prompt, prompt which we are not consuming. So input buffer has extra unconsumed prompt, which then can be consumed by any cmd creating descync on detecting when commands start and end.

[rvandepu]

This commit breaks config retrieval for me:

D, [2018-07-17T12:44:09.214153 #1823] DEBUG -- : lib/oxidized/job.rb: Starting fetching process for BCACTU01 at 2018-07-17 10:44:09 
D, [2018-07-17T12:44:10.429711 #1823] DEBUG -- : lib/oxidized/input/cli.rb: Running post_login commands at BCACTU01
D, [2018-07-17T12:44:10.429817 #1823] DEBUG -- : lib/oxidized/input/cli.rb: Running post_login command: nil, block: #<Proc:0x000000037fb6d0@/home/rvandepu/.config/oxidized/model/ios.rb:135> at BCACTU01
D, [2018-07-17T12:44:10.429893 #1823] DEBUG -- : Telnet: enable @BCACTU01
W, [2018-07-17T12:44:40.495561 #1823]  WARN -- : 192.168.249.206 raised Net::ReadTimeout (rescued Timeout::Error) with msg "timed out while waiting for more data"
D, [2018-07-17T12:44:40.495727 #1823] DEBUG -- : lib/oxidized/node.rb: Oxidized::Telnet failed for BCACTU01
D, [2018-07-17T12:44:40.495798 #1823] DEBUG -- : lib/oxidized/job.rb: Config fetched for BCACTU01 at 2018-07-17 10:44:40 UTC
W, [2018-07-17T12:44:41.222335 #1823]  WARN -- : BCACTU01 status no_connection, retries exhausted, giving up

But this is most probably, because what you said earlier. My config seems to use enable, while this shouldn't be used here.

My router.db file contains:

BCACTU01:IOS:-user-:-password-::telnet:BC

Config:

source:
  default: csv
  csv:
    file: /home/rvandepu/.config/oxidized/router.db
    delimiter: !ruby/regexp /:/
    map:
      name: 0
      model: 1
      username: 2
      password: 3
      input: 5
      group: 6
    vars_map:
      enable: 4
    gpg: false

Although the enable var is empty, it seems to be accepted by https://github.com/ytti/oxidized/blob/06705411e28cf747644beb78504c5d2a27713d1c/lib/oxidized/model/ios.rb#L136 How can I prevent this?

[ytti]

Change your enable from empty to true.

Now it expects to see enable prompt, and it doesn't so it breaks.

With true it wants to send enable, but expects normal prompt after.

[ytti]

You could also set enable to false, in which case enable is not sent at all, which you should have done in the first place, and the model would have worked.

Now you are telling Oxidizsed 'hey i need to enable on this node', which is essentially lie. The new model change allows new possibility 'hey i want need to enable on this node, but I actually won't be prompted for password.. Even the older one allowed you to say 'hey i don't need to enable on this this node', which is what you should have done, as that is the truth in your situation.

[rvandepu]

That is indeed what I was missing. Thank you very much for the fast responses.