donaldsharp
commented
6 years ago Can we get a show memory and a show thread cpu, show ip route and show run?
Closed ghost closed 6 years ago
donaldsharp
commented
6 years ago Can we get a show memory and a show thread cpu, show ip route and show run?
ghost
commented
6 years ago Kernel: 4.9.0-6-amd64 Distribution: Debian Stretch Memory: 8GB CPU: 4 vCores
free:
total used free shared buff/cache available
Mem: 8179256 8020884 120052 744 38320 1868
Swap: 8385532 7010152 1375380show thread cpu:
Thread statistics for zebra:
Showing statistics for pthread main
-----------------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
0 0.000 4 0 0 11 28 T if_zebra_speed_update
1 120.000 15 8000 48000 7994 47408 R zebra_accept
15 0.000 22 0 0 14 29 R zserv_read
1 0.000 2 0 0 18 21 R kernel_read
0 268.000 28 9571 12000 9547 12217 T work_queue_run
0 20.000 199 100 20000 104 19396 W zserv_write
3 0.000 6 0 0 41 44 T zebra_ptm_connect
0 368.000 22 16727 144000 16776 142100 E &zserv_process_messages
1 0.000 3 0 0 27 32 R vtysh_accept
1 0.000 19 0 0 39 104 R vtysh_read
Total thread statistics
-------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
22 776.000 320 2425 144000 2433 142100 RWTEX TOTAL
Thread statistics for ripd:
Showing statistics for pthread main
-----------------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
1 0.000 14 0 0 24 52 R zclient_read
1 0.000 3 0 0 30 44 R vtysh_accept
1 0.000 13 0 0 31 52 R vtysh_read
0 0.000 1 0 0 211 211 E zclient_connect
Total thread statistics
-------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
3 0.000 31 0 0 33 211 RWTEX TOTAL
Thread statistics for ripngd:
Showing statistics for pthread main
-----------------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
1 0.000 3 0 0 26 34 R vtysh_accept
1 0.000 13 0 0 35 57 R vtysh_read
1 0.000 14 0 0 23 47 R zclient_read
0 0.000 1 0 0 195 195 E zclient_connect
Total thread statistics
-------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
3 0.000 31 0 0 34 195 RWTEX TOTAL
Thread statistics for ospfd:
Showing statistics for pthread main
-----------------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
1 0.000 1 0 0 21 21 T ospf_lsa_refresh_walker
0 0.000 2 0 0 22 29 W ospf_write
1 0.000 3 0 0 28 36 R vtysh_accept
0 0.000 1 0 0 181 181 E zclient_connect
0 0.000 2 0 0 11 18 T ospf_spf_calculate_timer
0 0.000 1 0 0 1 1 E ospf_default_originate_timer
0 0.000 2 0 0 12 20 E ospf_ism_event
1 0.000 17 0 0 27 67 R zclient_read
1 4.000 2 2000 4000 60 84 R ospf_read
1 0.000 16 0 0 1011 15341 R vtysh_read
0 0.000 1 0 0 2 2 T ospf_sr_update_schedule
0 0.000 4 0 0 470 944 EX ospf_nsm_event
1 0.000 2 0 0 18 30 T (ospf_hello_timer)
0 0.000 1 0 0 35 35 T ospf_ase_calculate_timer
0 0.000 26 0 0 0 1 E ospf_external_lsa_originate_timer
Total thread statistics
-------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
6 4.000 81 49 4000 236 15341 RWTEX TOTAL
Thread statistics for ospf6d:
Showing statistics for pthread main
-----------------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
1 0.000 13 0 0 30 52 R vtysh_read
1 0.000 14 0 0 21 49 R zclient_read
1 0.000 3 0 0 23 36 R vtysh_accept
0 0.000 1 0 0 246 246 E zclient_connect
Total thread statistics
-------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
3 0.000 31 0 0 32 246 RWTEX TOTAL
Thread statistics for ldpd:
Showing statistics for pthread main
-----------------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
1 0.000 3 0 0 22 30 R vtysh_accept
0 0.000 27 0 0 41 902 W iev->handler_write
2 0.000 1 0 0 14 14 R iev_ldpe->handler_read
0 0.000 1 0 0 190 190 E zclient_connect
1 0.000 10 0 0 37 60 R vtysh_read
2 0.000 3 0 0 12 13 R iev_lde->handler_read
1 0.000 21 0 0 11 31 R zclient_read
Total thread statistics
-------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
7 0.000 66 0 0 31 902 RWTEX TOTAL
Thread statistics for bgpd:
Showing statistics for pthread main
-----------------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
1 0.000 3 0 0 35 65 R vtysh_accept
1 0.000 13 0 0 829 10165 R vtysh_read
0 0.000 1 0 0 181 181 T (bgp_start_timer)
2 0.000 18 0 0 7 19 R zclient_read
0 0.000 2 0 0 34 49 E zclient_connect
0 0.000 2 0 0 83 94 E bgp_event
Showing statistics for pthread BGP I/O thread
---------------------------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
Showing statistics for pthread BGP Keepalives thread
----------------------------------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
No data to display yet.
Total thread statistics
-------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
4 0.000 39 0 0 293 10165 RWTEX TOTAL
Thread statistics for isisd:
Showing statistics for pthread main
-----------------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
1 0.000 13 0 0 33 58 R vtysh_read
0 0.000 1 0 0 178 178 E zclient_connect
1 0.000 3 0 0 25 35 R vtysh_accept
1 0.000 14 0 0 21 37 R zclient_read
Total thread statistics
-------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
3 0.000 31 0 0 31 178 RWTEX TOTAL
Thread statistics for pimd:
Showing statistics for pthread main
-----------------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
1 0.000 13 0 0 34 57 R vtysh_read
1 0.000 3 0 0 28 33 R vtysh_accept
1 0.000 22 0 0 9 32 R zclient_read
1 0.000 1 0 0 20 20 T wheel_timer_thread
0 0.000 1 0 0 12 12 E zclient_lookup_connect
0 0.000 1 0 0 251 251 E zclient_connect
Total thread statistics
-------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
4 0.000 41 0 0 25 251 RWTEX TOTAL
Thread statistics for nhrpd:
Showing statistics for pthread main
-----------------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
1 0.000 13 0 0 32 58 R vtysh_read
1 0.000 7 0 0 47 53 T vici_reconnect
1 0.000 1 0 0 18 18 R netlink_route_recv
1 0.000 3 0 0 25 38 R vtysh_accept
0 0.000 1 0 0 3 3 T evmgr_reconnect
0 0.000 1 0 0 205 205 E zclient_connect
1 0.000 21 0 0 9 32 R zclient_read
Total thread statistics
-------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
5 0.000 47 0 0 26 205 RWTEX TOTAL
Thread statistics for eigrpd:
Showing statistics for pthread main
-----------------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
1 0.000 14 0 0 6 19 R zclient_read
1 0.000 3 0 0 28 36 R vtysh_accept
0 0.000 1 0 0 199 199 E zclient_connect
1 0.000 13 0 0 35 58 R vtysh_read
Total thread statistics
-------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
3 0.000 31 0 0 27 199 RWTEX TOTAL
Thread statistics for babeld:
Showing statistics for pthread main
-----------------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
1 0.000 14 0 0 15 34 R zclient_read
1 0.000 3 0 0 26 37 R vtysh_accept
1 0.000 13 0 0 34 52 R vtysh_read
0 0.000 1 0 0 238 238 E zclient_connect
Total thread statistics
-------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
3 0.000 31 0 0 32 238 RWTEX TOTAL
Thread statistics for watchfrr:
Showing statistics for pthread main
-----------------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
13 0.000 26 0 0 10 20 R handle_read
13 4.000 26 153 4000 33 87 T wakeup_send_echo
0 0.000 13 0 0 507 575 T wakeup_init
1 0.000 2 0 0 22 25 R vtysh_accept
0 0.000 6 0 0 27 38 R vtysh_read
Total thread statistics
-------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
27 4.000 73 54 4000 108 575 RWTEX TOTAL
Thread statistics for pbrd:
Showing statistics for pthread main
-----------------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
1 0.000 13 0 0 33 49 R vtysh_read
1 0.000 14 0 0 8 51 R zclient_read
1 0.000 3 0 0 23 35 R vtysh_accept
0 0.000 1 0 0 194 194 E zclient_connect
Total thread statistics
-------------------------
CPU (user+system): Real (wall-clock):
Active Runtime(ms) Invoked Avg uSec Max uSecs Avg uSec Max uSecs Type Thread
3 0.000 31 0 0 26 194 RWTEX TOTALshow memory:
Memory statistics for zebra:
System allocator statistics:
Total heap allocated: > 2GB
Holding block headers: 16 MiB
Used small blocks: 0 bytes
Used ordinary blocks: > 2GB
Free small blocks: 1952 bytes
Free ordinary blocks: 368 KiB
Ordinary blocks: 7739
Small blocks: 58
Holding blocks: 1
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer : 18 24
Buffer data : 1 4120
Host config : 2 (variably sized)
String vector : 3 (variably sized)
Command Tokens : 3370 72
Command Token Text : 2508 (variably sized)
Command Token Help : 2508 (variably sized)
Command Argument : 2 (variably sized)
Command Argument Name : 620 (variably sized)
Graph : 24 8
Graph Node : 3963 32
Hash : 122 (variably sized)
Hash Bucket : 1159684 32
Hash Index : 61 (variably sized)
Hook entry : 9 48
Interface : 4 248
Connected : 7 40
Link List : 41 40
Link Node : 117 24
Logging : 1 72
Temporary memory : 21 (variably sized)
Nexthop : 1264944 112
NetNS Context : 2 (variably sized)
NetNS Name : 1 18
Priority queue : 1 32
Priority queue data : 1 256
Prefix : 8 40
Privilege information : 2 (variably sized)
Stream : 1689253 40
Stream data : 1689253 (variably sized)
Stream FIFO : 30 24
Route table : 19 48
Route node : 1159073 (variably sized)
Thread : 38 (variably sized)
Thread master : 3 (variably sized)
Thread stats : 11 64
Vector : 7981 16
Vector index : 7981 (variably sized)
VRF : 1 184
VRF bit-map : 1269 (variably sized)
VTY : 6 (variably sized)
Work queue : 2 (variably sized)
Work queue item : 1 24
Work queue name string : 1 22
--- qmem Label Manager ---
Label Manager Chunk : 2 16
--- qmem zebra ---
ZEBRA VRF : 1 904
Route Entry : 632492 80
Static route : 1 192
RIB destination : 632480 48
RIB table info : 4 16
Nexthop tracking object : 4 192
Zebra Name Space : 1 312
--- qmem Table Manager ---
Memory statistics for ripd:
System allocator statistics:
Total heap allocated: 3148 KiB
Holding block headers: 0 bytes
Used small blocks: 0 bytes
Used ordinary blocks: 3038 KiB
Free small blocks: 1488 bytes
Free ordinary blocks: 110 KiB
Ordinary blocks: 1
Small blocks: 43
Holding blocks: 0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer : 3 24
Buffer data : 1 4120
Host config : 2 (variably sized)
String vector : 3 (variably sized)
Command Tokens : 1972 72
Command Token Text : 1568 (variably sized)
Command Token Help : 1568 (variably sized)
Command Argument : 2 (variably sized)
Command Argument Name : 318 (variably sized)
Graph : 18 8
Graph Node : 2383 32
Hash : 63 (variably sized)
Hash Bucket : 423 32
Hash Index : 32 (variably sized)
Hook entry : 2 48
Interface : 4 248
Connected : 7 40
Link List : 19 40
Link Node : 53 24
Logging : 1 72
Temporary memory : 2 (variably sized)
Priority queue : 1 32
Priority queue data : 1 256
Prefix : 8 40
Privilege information : 2 (variably sized)
Stream : 2 40
Stream data : 2 16384
Route table : 2 48
Thread : 7 (variably sized)
Thread master : 3 (variably sized)
Thread stats : 5 64
Vector : 4811 16
Vector index : 4811 (variably sized)
VRF : 1 184
VRF bit-map : 82 (variably sized)
VTY : 6 (variably sized)
Zclient : 1 2872
Redistribution instance IDs : 3 2
--- qmem ripd ---
RIP interface : 4 136
Memory statistics for ripngd:
System allocator statistics:
Total heap allocated: 3020 KiB
Holding block headers: 0 bytes
Used small blocks: 0 bytes
Used ordinary blocks: 2914 KiB
Free small blocks: 1472 bytes
Free ordinary blocks: 106 KiB
Ordinary blocks: 1
Small blocks: 43
Holding blocks: 0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer : 3 24
Buffer data : 1 4120
Host config : 2 (variably sized)
String vector : 3 (variably sized)
Command Tokens : 1662 72
Command Token Text : 1302 (variably sized)
Command Token Help : 1302 (variably sized)
Command Argument : 2 (variably sized)
Command Argument Name : 257 (variably sized)
Graph : 16 8
Graph Node : 1998 32
Hash : 57 (variably sized)
Hash Bucket : 348 32
Hash Index : 29 (variably sized)
Hook entry : 2 48
Interface : 8 (variably sized)
Connected : 7 40
Link List : 18 40
Link Node : 50 24
Logging : 1 72
Temporary memory : 2 (variably sized)
Priority queue : 1 32
Priority queue data : 1 256
Prefix : 8 40
Privilege information : 2 (variably sized)
Stream : 2 40
Stream data : 2 16384
Route table : 1 48
Thread : 7 (variably sized)
Thread master : 3 (variably sized)
Thread stats : 5 64
Vector : 4037 16
Vector index : 4037 (variably sized)
VRF : 1 184
VRF bit-map : 82 (variably sized)
VTY : 6 (variably sized)
Zclient : 1 2872
Redistribution instance IDs : 3 2
--- qmem ripngd ---
Memory statistics for ospfd:
System allocator statistics:
Total heap allocated: 4568 KiB
Holding block headers: 0 bytes
Used small blocks: 0 bytes
Used ordinary blocks: 4463 KiB
Free small blocks: 1520 bytes
Free ordinary blocks: 105 KiB
Ordinary blocks: 2
Small blocks: 43
Holding blocks: 0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer : 3 24
Buffer data : 1 4120
Host config : 2 (variably sized)
String vector : 3 (variably sized)
Command Tokens : 3393 72
Command Token Text : 2429 (variably sized)
Command Token Help : 2429 (variably sized)
Command Argument : 2 (variably sized)
Command Argument Name : 462 (variably sized)
Graph : 16 8
Graph Node : 3971 32
Hash : 348 (variably sized)
Hash Bucket : 679 32
Hash Index : 174 (variably sized)
Hook entry : 2 48
Interface : 4 248
Connected : 7 40
Link List : 68 40
Link Node : 237 24
Logging : 1 72
Temporary memory : 2 (variably sized)
Priority queue : 1 32
Priority queue data : 1 256
Prefix : 8 40
Privilege information : 2 (variably sized)
Stream : 4 40
Stream data : 4 (variably sized)
Route table : 147 48
Route node : 66 96
Thread : 37 (variably sized)
Thread master : 3 (variably sized)
Thread stats : 25 64
Vector : 7981 16
Vector index : 7981 (variably sized)
VRF : 1 184
VRF bit-map : 83 (variably sized)
VTY : 6 (variably sized)
Zclient : 1 2872
Redistribution instance IDs : 3 2
--- qmem ospfd ---
OSPF top : 1 3624
OSPF area : 1 248
OSPF network : 1 8
OSPF interface : 2 288
OSPF neighbor : 3 1472
OSPF route : 11 56
OSPF LSA : 13 112
OSPF LSA data : 13 (variably sized)
OSPF LSDB : 2 408
OSPF packet : 1 24
OSPF FIFO queue : 1 24
OSPF path : 11 16
OSPF ext. info : 3 48
OSPF if info : 4 32
OSPF if params : 4 104
OSPF MPLS parameters : 4 248
OSPF Extended parameters : 4 136
OSPF opaque function table : 3 112
OSPF External route table : 1 16
OSPF Redistriute : 1 32
Memory statistics for ospf6d:
System allocator statistics:
Total heap allocated: 3372 KiB
Holding block headers: 0 bytes
Used small blocks: 0 bytes
Used ordinary blocks: 3264 KiB
Free small blocks: 1472 bytes
Free ordinary blocks: 108 KiB
Ordinary blocks: 1
Small blocks: 43
Holding blocks: 0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer : 3 24
Buffer data : 1 4120
Host config : 2 (variably sized)
String vector : 3 (variably sized)
Command Tokens : 2670 72
Command Token Text : 1992 (variably sized)
Command Token Help : 1992 (variably sized)
Command Argument : 2 (variably sized)
Command Argument Name : 354 (variably sized)
Graph : 16 8
Graph Node : 3153 32
Hash : 52 (variably sized)
Hash Bucket : 496 32
Hash Index : 26 (variably sized)
Interface : 4 248
Connected : 7 40
Link List : 16 40
Link Node : 47 24
Logging : 1 72
Temporary memory : 2 (variably sized)
Priority queue : 1 32
Priority queue data : 1 256
Prefix : 8 40
Privilege information : 2 (variably sized)
Stream : 2 40
Stream data : 2 16384
Thread : 8 (variably sized)
Thread master : 3 (variably sized)
Thread stats : 6 64
Vector : 6346 16
Vector index : 6346 (variably sized)
VRF : 1 184
VRF bit-map : 82 (variably sized)
VTY : 6 (variably sized)
Zclient : 1 2872
Redistribution instance IDs : 3 2
--- qmem ospf6d ---
Memory statistics for ldpd:
System allocator statistics:
Total heap allocated: 3320 KiB
Holding block headers: 0 bytes
Used small blocks: 0 bytes
Used ordinary blocks: 3192 KiB
Free small blocks: 1504 bytes
Free ordinary blocks: 128 KiB
Ordinary blocks: 2
Small blocks: 44
Holding blocks: 0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer : 3 24
Buffer data : 1 4120
Host config : 2 (variably sized)
String vector : 3 (variably sized)
Command Tokens : 1732 72
Command Token Text : 1336 (variably sized)
Command Token Help : 1336 (variably sized)
Command Argument : 2 (variably sized)
Command Argument Name : 350 (variably sized)
Graph : 18 8
Graph Node : 2064 32
Hash : 40 (variably sized)
Hash Bucket : 348 32
Hash Index : 20 (variably sized)
Interface : 4 248
Connected : 7 40
Link List : 16 40
Link Node : 35 24
Logging : 1 72
Temporary memory : 2 (variably sized)
Priority queue : 1 32
Priority queue data : 1 256
Prefix : 8 40
Privilege information : 2 (variably sized)
Stream : 2 40
Stream data : 2 16384
Thread : 11 (variably sized)
Thread master : 3 (variably sized)
Thread stats : 8 64
Vector : 4169 16
Vector index : 4169 (variably sized)
VRF : 1 184
VRF bit-map : 82 (variably sized)
VTY : 6 (variably sized)
Zclient : 1 2872
Redistribution instance IDs : 3 2
Memory statistics for bgpd:
System allocator statistics:
Total heap allocated: 542 MiB
Holding block headers: 16 MiB
Used small blocks: 0 bytes
Used ordinary blocks: 536 MiB
Free small blocks: 92 KiB
Free ordinary blocks: 5867 KiB
Ordinary blocks: 169
Small blocks: 1170
Holding blocks: 1
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer : 4 24
Buffer data : 557 4120
Host config : 2 (variably sized)
String vector : 3 (variably sized)
Command Tokens : 9102 72
Command Token Text : 6779 (variably sized)
Command Token Help : 6779 (variably sized)
Command Argument : 2 (variably sized)
Command Argument Name : 1426 (variably sized)
FRR POSIX Thread : 4 (variably sized)
POSIX synchronization primitives: 4 (variably sized)
Graph : 35 8
Graph Node : 10864 32
Hash : 370 (variably sized)
Hash Bucket : 1475225 32
Hash Index : 186 (variably sized)
Interface : 4 248
Connected : 7 40
Link List : 38 40
Link Node : 220 24
Logging : 1 72
Temporary memory : 5 (variably sized)
Nexthop : 3 112
Priority queue : 3 32
Priority queue data : 3 256
Prefix : 8 40
Privilege information : 2 (variably sized)
Ring buffer : 4 (variably sized)
Skip List : 2 56
Skip Node : 4 160
Socket union : 3 28
Stream : 11 40
Stream data : 11 (variably sized)
Stream FIFO : 4 24
Route table : 106 48
Thread : 23 (variably sized)
Thread master : 11 (variably sized)
Thread stats : 23 64
Vector : 21805 16
Vector index : 21805 (variably sized)
VRF : 1 184
VRF bit-map : 161 (variably sized)
VTY : 6 (variably sized)
Work queue : 3 144
Work queue name string : 3 (variably sized)
Zclient : 2 2872
Redistribution instance IDs : 6 2
--- qmem rfapi ---
NVE Configuration : 1 2560
RFAPI Generic : 1 296
RFAPI Import Table : 1 208
--- qmem bgpd ---
BGP Label FIFO : 1 48
BGP PBR Context : 1 16
BGP instance : 2 (variably sized)
BGP listen socket details : 2 48
BGP peer : 3 20688
BGP peer hostname : 4 (variably sized)
BGP peer af : 1 80
BGP update group : 1 104
BGP update subgroup : 1 240
BGP packet : 1 56
BGP attribute : 107420 224
BGP aspath : 98124 40
BGP aspath seg : 98299 24
BGP aspath segment data : 98299 (variably sized)
BGP aspath str : 98124 (variably sized)
BGP table : 93 40
BGP node : 1267816 152
BGP route : 693148 112
BGP ancillary route info : 693148 200
BGP connected : 5 4
BGP synchronise : 43 72
community : 24 40
community val : 24 (variably sized)
community str : 24 (variably sized)
extcommunity : 1 32
extcommunity val : 1 (variably sized)
extcommunity str : 1 (variably sized)
community-list handler : 1 96
BGP transit attr : 4 24
BGP transit val : 4 (variably sized)
BGP nexthop : 3 72
BGP own address : 3 8
Large Community : 32 40
Large Community display string: 32 (variably sized)
Large Community value : 32 (variably sized)
Memory statistics for isisd:
System allocator statistics:
Total heap allocated: 3216 KiB
Holding block headers: 0 bytes
Used small blocks: 0 bytes
Used ordinary blocks: 3111 KiB
Free small blocks: 1504 bytes
Free ordinary blocks: 105 KiB
Ordinary blocks: 1
Small blocks: 44
Holding blocks: 0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer : 3 24
Buffer data : 1 4120
Host config : 2 (variably sized)
String vector : 3 (variably sized)
Command Tokens : 2171 72
Command Token Text : 1707 (variably sized)
Command Token Help : 1707 (variably sized)
Command Argument : 2 (variably sized)
Command Argument Name : 283 (variably sized)
Graph : 16 8
Graph Node : 2642 32
Hash : 52 (variably sized)
Hash Bucket : 488 32
Hash Index : 26 (variably sized)
Hook entry : 2 48
Interface : 4 248
Connected : 7 40
Link List : 34 40
Link Node : 57 24
Logging : 1 72
Temporary memory : 2 (variably sized)
Priority queue : 1 32
Priority queue data : 1 256
Prefix : 15 40
Privilege information : 2 (variably sized)
Stream : 2 40
Stream data : 2 16384
Thread : 8 (variably sized)
Thread master : 3 (variably sized)
Thread stats : 6 64
Vector : 5323 16
Vector index : 5323 (variably sized)
VRF : 1 184
VRF bit-map : 82 (variably sized)
VTY : 6 (variably sized)
Zclient : 1 2872
Redistribution instance IDs : 3 2
--- qmem isisd ---
ISIS : 1 160
ISIS circuit : 3 704
ISIS MPLS_TE parameters : 3 144
Memory statistics for pimd:
System allocator statistics:
Total heap allocated: 16 MiB
Holding block headers: 0 bytes
Used small blocks: 0 bytes
Used ordinary blocks: 16 MiB
Free small blocks: 1504 bytes
Free ordinary blocks: 110 KiB
Ordinary blocks: 1
Small blocks: 44
Holding blocks: 0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer : 4 24
Buffer data : 1 4120
Host config : 2 (variably sized)
String vector : 3 (variably sized)
Command Tokens : 2654 72
Command Token Text : 2046 (variably sized)
Command Token Help : 2046 (variably sized)
Command Argument : 2 (variably sized)
Command Argument Name : 364 (variably sized)
Graph : 16 8
Graph Node : 3243 32
Hash : 64 (variably sized)
Hash Bucket : 605 32
Hash Index : 32 (variably sized)
Interface : 4 248
Connected : 7 40
Link List : 122 40
Link Node : 54 24
Logging : 1 72
Temporary memory : 2 (variably sized)
Priority queue : 1 32
Priority queue data : 1 256
Prefix : 8 40
Privilege information : 2 (variably sized)
Stream : 4 40
Stream data : 4 16384
Route table : 1 48
Route node : 1 96
Thread : 9 (variably sized)
Thread master : 3 (variably sized)
Thread stats : 8 64
Vector : 6525 16
Vector index : 6525 (variably sized)
VRF : 1 184
VRF bit-map : 108 (variably sized)
VTY : 6 (variably sized)
Timer Wheel : 1 72
Timer Wheel Slot List : 1 800
Zclient : 2 2872
Redistribution instance IDs : 3 2
--- qmem pimd ---
PIM RP info : 1 176
PIM global state : 1 544
PIM SSM configuration : 1 8
Memory statistics for nhrpd:
System allocator statistics:
Total heap allocated: 28 MiB
Holding block headers: 2052 KiB
Used small blocks: 0 bytes
Used ordinary blocks: 27 MiB
Free small blocks: 1536 bytes
Free ordinary blocks: 463 KiB
Ordinary blocks: 2
Small blocks: 45
Holding blocks: 1
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer : 3 24
Buffer data : 1 4120
Host config : 2 (variably sized)
String vector : 3 (variably sized)
Command Tokens : 1157 72
Command Token Text : 927 (variably sized)
Command Token Help : 927 (variably sized)
Command Argument : 2 (variably sized)
Command Argument Name : 134 (variably sized)
Graph : 12 8
Graph Node : 1387 32
Hash : 42 (variably sized)
Hash Bucket : 134360 32
Hash Index : 21 (variably sized)
Hook entry : 2 48
Interface : 4 248
Connected : 7 40
Link List : 16 40
Link Node : 38 24
Logging : 1 72
Temporary memory : 2 (variably sized)
Priority queue : 1 32
Priority queue data : 1 256
Prefix : 8 40
Privilege information : 2 (variably sized)
Stream : 2 40
Stream data : 2 16384
Route table : 4 48
Route node : 134114 96
Thread : 10 (variably sized)
Thread master : 3 (variably sized)
Thread stats : 9 64
Vector : 2805 16
Vector index : 2805 (variably sized)
VRF : 1 184
VRF bit-map : 82 (variably sized)
VTY : 6 (variably sized)
Zclient : 1 2872
Redistribution instance IDs : 3 2
--- qmem NHRP ---
NHRP interface : 4 424
NHRP routing entry : 69917 48
Memory statistics for eigrpd:
System allocator statistics:
Total heap allocated: 3152 KiB
Holding block headers: 0 bytes
Used small blocks: 0 bytes
Used ordinary blocks: 3044 KiB
Free small blocks: 1472 bytes
Free ordinary blocks: 108 KiB
Ordinary blocks: 1
Small blocks: 43
Holding blocks: 0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer : 3 24
Buffer data : 1 4120
Host config : 2 (variably sized)
String vector : 3 (variably sized)
Command Tokens : 2029 72
Command Token Text : 1605 (variably sized)
Command Token Help : 1605 (variably sized)
Command Argument : 2 (variably sized)
Command Argument Name : 319 (variably sized)
Graph : 18 8
Graph Node : 2417 32
Hash : 57 (variably sized)
Hash Bucket : 400 32
Hash Index : 29 (variably sized)
Hook entry : 1 48
Interface : 4 248
Connected : 7 40
Link List : 18 40
Link Node : 50 24
Logging : 1 72
Temporary memory : 2 (variably sized)
Priority queue : 1 32
Priority queue data : 1 256
Prefix : 8 40
Privilege information : 2 (variably sized)
Stream : 2 40
Stream data : 2 16384
Thread : 7 (variably sized)
Thread master : 3 (variably sized)
Thread stats : 5 64
Vector : 4877 16
Vector index : 4877 (variably sized)
VRF : 1 184
VRF bit-map : 82 (variably sized)
VTY : 6 (variably sized)
Zclient : 1 2872
Redistribution instance IDs : 3 2
--- qmem eigrpd ---
Memory statistics for babeld:
% [BABELD] Unknown command: do show memory
Memory statistics for watchfrr:
System allocator statistics:
Total heap allocated: 264 KiB
Holding block headers: 0 bytes
Used small blocks: 0 bytes
Used ordinary blocks: 204 KiB
Free small blocks: 1328 bytes
Free ordinary blocks: 60 KiB
Ordinary blocks: 1
Small blocks: 38
Holding blocks: 0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer : 1 24
Buffer data : 1 4120
Host config : 1 9
String vector : 3 (variably sized)
Command Tokens : 343 72
Command Token Text : 285 (variably sized)
Command Token Help : 285 (variably sized)
Command Argument : 2 (variably sized)
Command Argument Name : 36 (variably sized)
Graph : 6 8
Graph Node : 436 32
Hash : 16 (variably sized)
Hash Bucket : 102 32
Hash Index : 8 (variably sized)
Link List : 5 40
Link Node : 11 24
Logging : 1 72
Temporary memory : 2 (variably sized)
Priority queue : 1 32
Priority queue data : 1 256
Thread : 31 (variably sized)
Thread master : 3 (variably sized)
Thread stats : 7 64
Vector : 889 16
Vector index : 889 (variably sized)
VTY : 3 (variably sized)
Memory statistics for pbrd:
System allocator statistics:
Total heap allocated: 2820 KiB
Holding block headers: 0 bytes
Used small blocks: 0 bytes
Used ordinary blocks: 2713 KiB
Free small blocks: 1472 bytes
Free ordinary blocks: 107 KiB
Ordinary blocks: 1
Small blocks: 43
Holding blocks: 0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer : 3 24
Buffer data : 1 4120
Host config : 2 (variably sized)
String vector : 3 (variably sized)
Command Tokens : 1146 72
Command Token Text : 898 (variably sized)
Command Token Help : 898 (variably sized)
Command Argument : 2 (variably sized)
Command Argument Name : 169 (variably sized)
Graph : 14 8
Graph Node : 1378 32
Hash : 36 (variably sized)
Hash Bucket : 244 32
Hash Index : 18 (variably sized)
Interface : 4 248
Connected : 7 40
Link List : 16 40
Link Node : 35 24
Logging : 1 72
Temporary memory : 2 (variably sized)
Priority queue : 1 32
Priority queue data : 1 256
Prefix : 8 40
Privilege information : 2 (variably sized)
Stream : 2 40
Stream data : 2 16384
Thread : 7 (variably sized)
Thread master : 3 (variably sized)
Thread stats : 5 64
Vector : 2789 16
Vector index : 2789 (variably sized)
VRF : 1 184
VRF bit-map : 82 (variably sized)
VTY : 6 (variably sized)
Zclient : 1 2872
Redistribution instance IDs : 3 2
--- qmem pbrd ---
PBR Interface : 4 100
`
relevant config:
`
router bgp xxx
neighbor x.x.x.x remote-as internal
neighbor x.x.x.x update-source x.x.x.y
!
router ospf
redistribute connected
network x/24 area 0
!I won't be posting 700k lines (for show ip route) on here.
donaldsharp
commented
6 years ago
Stream data : 1689253 (variably sized)```
I would suspect we have a leak in stream memory. InvestigatingThanks. FRR always used slightly more memory than bird (around ~400MB for a full table with fib export on bird2, around ~ 1.5GB for that on frr) but suddenly allocating a steady 8GB can't be right.
donaldsharp
commented
6 years ago some more commands to run ss -aep | grep frr, show workqueues, does zebra have any error messages in it's logs?
donaldsharp
commented
6 years ago I have a recreate:
30580 frr 15 -5 10.956g 3.123g 3344 S 0.0 13.3 0:13.54 zebra
30587 frr 15 -5 709892 520416 4308 S 0.0 2.1 0:07.52 bgpd
30596 frr 15 -5 73956 8792 2700 S 0.0 0.0 0:03.33 pimd
30603 frr 15 -5 59308 4276 2468 S 0.0 0.0 0:00.00 pbrd
mwinter-osr
commented
6 years ago I've just set up a FRR 5.0 (RPKI package version for debian 9 as published on the github release page). The issue does not show up in my setup. I have 1 full BGP feed receive and 1 full BGP feed to advertise (IPv4 only, no IPv6) Memory after startup is similar to 4.0 RPKI version as before.
Does this happen immediatly? Can you post config and exact version (ie "show version" if built on your own, otherwise where you got the package)
donaldsharp
commented
6 years ago @sybevnao -> If you have the ability to try this branch, I believe this will fix your issue.
mwinter-osr
commented
6 years ago @sybevnao If you have a chance to test the fix from Donald, then feel free to use the packages here: https://ci1.netdef.org/browse/FRR-FRRPULLREQ-4153/artifact
These are the packages built based on Donald's fix. Feedback would be very welcome.
ghost
commented
6 years ago I've just upgraded the frr package using the packages provided by @mwinter-osr, after successfully installing & restarting all services, massive decrease in memory usage can be observed:
Linux "free"
total used free shared buff/cache available
Mem: 8179256 1661624 6375204 8664 142428 6305112
Swap: 8385532 0 8385532zebra
Memory statistics for zebra:
System allocator statistics:
Total heap allocated: 777 MiB
Holding block headers: 16 MiB
Used small blocks: 0 bytes
Used ordinary blocks: 508 MiB
Free small blocks: 1968 bytes
Free ordinary blocks: 269 MiB
Ordinary blocks: 100229
Small blocks: 58
Holding blocks: 1bgpd
Memory statistics for bgpd:
System allocator statistics:
Total heap allocated: 564 MiB
Holding block headers: 16 MiB
Used small blocks: 0 bytes
Used ordinary blocks: 534 MiB
Free small blocks: 8120 KiB
Free ordinary blocks: 30 MiB
Ordinary blocks: 14430
Small blocks: 103156
Holding blocks: 1BGP summary
IPv4 Unicast Summary:
BGP router identifier x, local AS number x vrf-id 0
BGP table version 693917
RIB entries 1266982, using 184 MiB of memory
Peers 1, using 21 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
x 4 x 137753 6 0 0 0 00:03:58 692745
Total number of neighbors 1@donaldsharp indeed seems to have spot the culprit and current usage is more than reasonable.
Thanks to @donaldsharp and @mwinter-osr for your effort and the super-quick responses and follow-ups!
donaldsharp
commented
6 years ago @sybevnao -> If you are not using pim, then shutting it off would reduce some memory usage in zebra. ( See https://github.com/FRRouting/frr/pull/2532/commits/6298b1f5c974b0de8885c6dd5f69c40f132f8da4 ) that I just filed.
ghost
commented
6 years ago @donaldsharp there indeed seems to be a bit of an improvement after setting pimd=no:
zebra
Total heap allocated: 597 MiBAdditionally, I've just taken a look at the CPU usage of the entire system now, which seems to outperform BIRD on a system with similar specifications now:
frr
%Cpu(s): 0.1 us, 0.2 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 stbird
%Cpu(s): 14.7 us, 8.7 sy, 0.0 ni, 76.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.3 stGood job. My hypothesis here is that frr deals better with occasional bgp updates.
donaldsharp
commented
6 years ago @sybevnao good to hear. I believe we still have some serious room for improvements ;)
Could anyone here explain why the memory usage of the version 5 release is literally crazy? I'm exporting one full table to a virtual machine with 8GB of memory and the entire memory is filled by that from zebra, including a fair amount of swap as well.
Memory usage for a full table used to be below 4GB in older releases.
What's the point + reason for this increase? Makes any usage of frr useless and turns anyone who doesn't want to literally and unnecessarily waste memory use bird.