search

derek-shnosh / ztp-watcher

Supplement to FreeZTP service to help with switch upgrades.
MIT License
9 stars 3 forks source link

(Nornir?) SSH connection crashes 2960L #2

Open derek-shnosh opened 5 years ago

derek-shnosh commented 5 years ago

Problem/Symptom

The SSH connection test succeeds which kicks off the nornir connection to transfer the image via TFTP, the switch crashes shortly thereafter.

Details

2960L-16TS-LL switch running c2960l-universalk9-mz.152-6.E2b.bin

Switch log 

General initialization - Version: 1.0.0
Serdes initialization - Version: 1.0.2
Board H/W SKU Id is (2)
DDR3 Training Sequence - Ver TIP-1.27.0
DDR3 Training Sequence - Switching XBAR Window to FastPath Window 
DDR3 Training Sequence - Ended Successfully
DDR3 Training Sequence - Start scrubbing 
DDR3 Training Sequence - End scrubbing 
BootROM: Image checksum verification PASSED

Board SKU ID is (2)
FPU initialized to Run Fast Mode.
FPGA: 1.5
DRAM Size: 512 MB
Xmodem file system is available.
USB EHCI 1.00
Using driver version 1 for media type 3
yaffs[9]: Initialization complete.

yaffs[10]: Initialization complete.

yaffs[11]: Initialization complete.

Base ethernet MAC Address: 78:02:b1:9d:d2:00
The password-recovery mechanism is enabled.

*** The system will autoboot in 5 seconds ***
Send break character to prevent autobooting.

....................
config file not found..
Loading "flash:c2960l-universalk9-mz.152-6.E2b/c2960l-universalk9-mz.152-6.E2b.bin"...Verifying image flash:c2960l-universalk9-mz.152-6.E2b/c2960l-universalk9-mz.152-6.E2b.bin...........................................................................................................................................................................................................................................
Image passed digital signature verification
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
File "flash:c2960l-universalk9-mz.152-6.E2b/c2960l-universalk9-mz.152-6.E2b.bin" uncompressed and installed, entry point: 0x3000
executing...

              Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

           cisco Systems, Inc.
           170 West Tasman Drive
           San Jose, California 95134-1706

Cisco IOS Software, C2960L Software (C2960L-UNIVERSALK9-M), Version 15.2(6)E2b, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Tue 20-Nov-18 21:04 by prod_rel_team
Initializing flash...
### Found Flash Chip id - 0xDA2C
Using driver version 1 for media type 3
yaffs[8]: Initialization complete.

yaffs[9]: Initialization complete.

yaffs[10]: Initialization complete.

...done Initializing flash.

Checking for MCU firmware upgrade..
MCU firmware upgrade not needed (v1.6.1.iap)

Checking for Bootloader upgrade..
New version = Version 15.2(5r)E3            , 
Current version = Version 15.2(6r)E, RELEASE SOF

Boot Loader upgrade not needed(v) DB Recovery 4

INIT: Pp init completed.

INIT: ASIC 0 phases init completed.

INIT: phases init completed.
INIT: features init completed.

POST: ACT2 Authentication : Begin
POST: ACT2 Authentication : End, Status Passed
POST: Thermal Tests : Begin
POST: Thermal Tests : End, Status Passed

POST: PortASIC Port Loopback Tests : Begin
POST: PortASIC Port Loopback Tests : End, Status Passed

Initializing Device Manager......done
Initializing OBFL Module......done

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2960L-16TS-LL (Marvell PJ4B (584) v7 (Rev 2)) processor (revision F0) with 524288K bytes of memory.
Processor board ID FCW2318A2SG
Last reset from Reload
1 Virtual Ethernet interface
18 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 78:02:B1:9D:D2:00
Motherboard assembly number     : U58O342T04
Power supply part number        : 341-100559-01
Motherboard serial number       : FOC23180JV8
Power supply serial number      : DCG22501L3X
Model revision number           : F0
Motherboard revision number     : 24
Model number                    : WS-C2960L-16TS-LL
System serial number            : FCW2318A2SG
Top Assembly Part Number        : 74-105853-02
Top Assembly Revision Number    : D0
Version ID                      : V02
CLEI Code Number                : CMMWG00ARB
Hardware Board Revision Number  : 0x01

         --- System Configuration Dialog ---

Enable secret warning
----------------------------------
In order to access the device manager, an enable secret is required
If you enter the initial configuration dialog, you will be prompted for the enable secret
If you choose not to enter the intial configuration dialog, or if you exit setup without setting the enable secret,
please set an enable secret using the following CLI in configuration mode-
enable secret 0 <cleartext password>
----------------------------------
Would you like to enter the initial configuration dialog? [yes/no]: 
Loading network-confg from 172.17.251.251 (via Vlan1): !
[OK - 72 bytes]

Loading ZTP-2489931C9A-confg from 172.17.251.251 (via Vlan1): !
[OK - 9821 bytes]
The name for the keys will be: hostname.domain.local

% The key modulus size is 2048 bits
% Generating 2048 bit RSA keys, keys will be non-exportable...
Sep 23 20:10:43.255: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.200.128.240 port 514 started - CLI initiated
Sep 23 20:10:43.256: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.200.136.241 port 514 started - CLI initiated
Sep 23 2019 14:10:44.415 CT: %SYS-6-CLOCKUPDATE: System clock has been updated from 20:10:44 UTC Mon Sep 23 2019 to 14:10:44 CT Mon Sep 23 2019, configured from console by console.
Sep 23 2019 15:10:44.417 CDT: %SYS-6-CLOCKUPDATE: System clock has been updated from 14:10:44 CT Mon Sep 23 2019 to 15:10:44 CDT Mon Sep 23 2019, configured from console by console.
Sep 23 2019 15:10:44.424 CDT: %SYS-5-LOG_CONFIG_CHANGE: Buffer logging: level debugging, xml disabled, filtering disabled, size (409600)
Sep 23 2019 15:10:44.427 CDT: %SYS-5-LOG_CONFIG_CHANGE: Console logging: level debugging, xml disabled, filtering disabled
Sep 23 2019 15:10:44.434 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
Sep 23 2019 15:10:46.401 CDT: %LINK-5-CHANGED: Interface GigabitEthernet0/17, changed state to administratively down
[OK] (elapsed time was 10 seconds)

Sep 23 2019 15:10:54.064 CDT: %SSH-5-ENABLED: SSH 1.99 has been enabled

Press RETURN to get started!

Sep 23 2019 15:11:01.241 CDT: %RADIUS-4-RADIUS_ALIVE: RADIUS server 10.200.16.10:1812,1813 is being marked alive.
Sep 23 2019 15:11:01.241 CDT: %RADIUS-4-RADIUS_ALIVE: RADIUS server 10.200.32.10:1812,1813 is being marked alive.
Sep 23 2019 15:11:01.241 CDT: %RADIUS-4-RADIUS_ALIVE: RADIUS server 10.200.16.8:1812,1813 is being marked alive.
Sep 23 2019 15:11:01.321 CDT: %SYS-5-CONFIG_I: Configured from tftp://172.17.251.251/ZTP-2489931C9A-confg by console
Sep 23 2019 15:11:03.193 CDT: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down
Sep 23 2019 15:11:31.318 CDT: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1134, changed state to up
Sep 23 2019 15:11:35.983 CDT: %PNP-6-HTTP_CONNECTING: PnP Discovery trying to connect to PnP server https://18.205.167.7:443/pnp/HELLO
Sep 23 2019 15:11:36.026 CDT: %PNP-6-HTTP_CONNECTED: PnP Discovery connected to PnP server https://18.205.167.7:443/pnp/HELLO
Sep 23 2019 15:11:38.865 CDT: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.200.136.241 port 514 started - reconnection
Sep 23 2019 15:11:43.865 CDT: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.200.128.240 port 514 started - reconnection

Unexpected exception to CPU vector 4 (data abort), PC = 19145FC 
-Traceback= 0x19145FCz 

Fault address:BEADE063

Registers:
R0 :BEADDEEF   R1 :044B71CC   R2 :044B71CE   R3 :01914FEC   R4 :BEADDEEF   
R5 :044B71CC   R6 :044B71CE   R7 :00000001   R8 :05332A90   R9 :054D3120   
R10:03C7069C   R11:044B719C   R12:00000000   R13:044B719C   R14:01914FF8   
R15:019145FC      PSR : A000031F
Exception Message : 

Fault Status(000010F8):External abort on non-linefetch (page), domain:15, FAR:BEADE063 MMU Entry:n/a

validblock_diagnose, code = 2

current memory block, bp = 0x6127074,
memorypool type is Processor
data check, ptr = 0x612709C
bp->next(0x218D98C) not in any mempool

previous memory block, bp = 0x6127104,
memorypool type is Processor
data check, ptr = 0x612712C
========= Dump bp = 0x6127074 ======================

 6126F74:        0        0        0        0 FD0110DF AB1234CD   EC0000       F1
 6126F94:  264D568    C4B24  6126FD8  6126F4C       14        0    BFA50 DEADBEEF
 6126FB4:    BFA50    3ABB0  D0D0D0D  611A4A0  60F7C50  D0D0D0D  D0D0D0D  D0D0D0D
 6126FD4:  D0D0D0D AB1234CD   EC0000       F1  2ABE654  11209B8  6127028  6126F9C
 6126FF4: 80000014        1    3AB98 444F5431 583A7375 62736372 69626572 3A310000
 6127014:        0        0        0        0 FD0110DF AB1234CD   EC0000       F1
 6127034:  264DD78    C64D0  6127078  6126FEC 80000014        1    BFA80  60FCC80
 6127054:        0        0        0        0        0        0        0        0
 6127074: FD0110DF AB1234CD   EC0000       F1  2DDBBE4  218D98C  6127118  612703C
 6127094: 8000003C        1    BFA80        1        0 73850001  61270A0  5C6C220
 61270B4:  4458A00    10000  34AC43C        0  61270A0  34AC5F4  5A8A3B0  1000000
 61270D4:  32BA184  5C55E48  3C2F180  55493C0        4 80500010  21C6CEC        0
 61270F4:        0        0        0        0        0        0        0        0
 6127114: FD0110DF AB1234CD   EC0000       F1  26238F8    3E1FC  6127168  612708C
 6127134: 80000014        1    3AB98        0        2 14001400        0        0
 6127154:        0        0        0        0 FD0110DF AB1234CD   EC0000       F1
========= Dump bp->next = 0x218D98C ======================

 218D88C: 2C300BE5  730A0E1  400A0E1  810A0E1  A20A0E1 DD83DEEB 28301BE5     53E3
 218D8AC:  20000DA  42196E7  20053E1  D0000BA     55E3  150A013  500001A 2C201BE5
 218D8CC:     52E3  20000DA 203099E5  30052E1  F0000BA  14084E2  30054E3 E8FFFF1A
 218D8EC: 20D04BE2 F08FBDE8 40104BE2 38309FE5 38209FE5 1400A0E3   408DE5 1FBB09EB
 218D90C: 40104BE2 28201BE5 24009FE5 4A608DEB E6FFFFEA 18009FE5  150A0E3 14109FE5
 218D92C: 45608DEB EAFFFFEA  390B190  2E4CE0C  298CB98  2DC7190  2C8CD50 F0492DE9
 218D94C: 18B08DE2 14D04DE2   40A0E1  160A0E1  82090E5  C3090E5 B882D1E1  33092E1
 218D96C: 2300000A   70A0E3  100A0E3 5810A0E3 EC209FE5  730A0E1   708DE5 CE61D1EB
 218D98C:   5050E2 3000000A  710A0E1 5820A0E3 C79F0DEB  500A0E1 384085E5 3C6085E5
 218D9AC: 19D500EB BC309FE5 101085E2 20204BE2 A90F84E2 859B09EB AF0F84E2  810A0E1
 218D9CC: F1B20DEB 3C3094E5     53E3  E00001A 203094E5  110A0E3 100093E5 3EFFFFEB
 218D9EC:  400A0E1 BF21FFEB  500A0E1 18D04BE2 F089BDE8 70009FE5 271400E3 6C209FE5
 218DA0C: 6C309FE5 5522FFEB D5FFFFEA 440084E2 B812D6E1 DCB20DEB 643294E5     53E3
 218DA2C: EAFFFF1A 787294E5     57E3 E7FFFF0A 440087E2 B812D6E1 D3B20DEB 787297E5
 218DA4C:     57E3 F9FFFF1A E0FFFFEA 24009FE5 2A1400E3 14209FE5 14309FE5 3F22FFEB
 218DA6C: E0FFFFEA  2DDBBE4  218B354  2DDBBCC  2DDB7F4  2DC9140  2DE1958 F0492DE9
========== Dump bp->previous = 0x6127118 =====================

 6127018:        0        0        0 FD0110DF AB1234CD   EC0000       F1  264DD78
 6127038:    C64D0  6127078  6126FEC 80000014        1    BFA80  60FCC80        0
 6127058:        0        0        0        0        0        0        0 FD0110DF
 6127078: AB1234CD   EC0000       F1  2DDBBE4  218D98C  6127118  612703C 8000003C
 6127098:        1    BFA80        1        0 73850001  61270A0  5C6C220  4458A00
 61270B8:    10000  34AC43C        0  61270A0  34AC5F4  5A8A3B0  1000000  32BA184
 61270D8:  5C55E48  3C2F180  55493C0        4 80500010  21C6CEC        0        0
 61270F8:        0        0        0        0        0        0        0 FD0110DF
 6127118: AB1234CD   EC0000       F1  26238F8    3E1FC  6127168  612708C 80000014
 6127138:        1    3AB98        0        2 14001400        0        0        0
 6127158:        0        0        0 FD0110DF AB1234CD   EC0000       F1  2623CDC
 6127178:    3F050  61271B8  612712C 80000014        1    3A8CC        9        0
 6127198:        0        0        0        0        0        0        0 FD0110DF
 61271B8: AB1234CD   EC0000       F1  262386C    3E018  6127208  612717C 80000014
 61271D8:        1    3A8DC       12        0        0        0        0        0
 61271F8:        0        0        0 FD0110DF AB1234CD        0        0  2BD51D8
============================================

%Software-forced reload

flash0:/ppoe.bin: No such devicedrgWriteToStorage Failed]

eneral initialization - Version: 1.0.0
Serdes initialization - Version: 1.0.2
Board H/W SKU Id is (2)
DDR3 Training Sequence - Ver TIP-1.27.0
helpdeskdan commented 5 years ago

So, malformed ssh packet can crash 152-6.E2b? My interest is piqued.