search

FRRouting / frr

The FRRouting Protocol Suite
https://frrouting.org/
Other
3.35k stars 1.25k forks source link

Arbitarily high memory usage (again?) #3283

Closed ghost closed 2 years ago

ghost commented 6 years ago

Already had a similar issue a while ago: #2527

On a system working as route reflector which is part of an IS-IS area (the reflector is deployed to connect a bunch of Junipers), there is currently 3 BGP full feeds:

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
10.42.42.9      4      49004  734313 2009594        0    0    0 00:53:54       748336
10.42.42.34     4      49004  963578 3470979        0    0    0 00:53:58       748335
10.42.42.35     4      49004  588650    1450        0    0    0 00:58:36       714935

Memory usage for the RIB according to show bgp summary:

BGP router identifier 10.42.42.40, local AS number 49004 vrf-id 0
BGP table version 40314070
RIB entries 1367056, using 209 MiB of memory
Peers 4, using 82 KiB of memory
Peer groups 1, using 64 bytes of memory

now, taking a look at the systems memory:

              total        used        free      shared  buff/cache   available
Mem:           1996        1897          71           0          26          11
Swap:          2047        1484         563

this is a bit far off sanity. The same full tables barely take a GB of memory on juniper.

Now checking the show memory command:

Memory statistics for zebra:
System allocator statistics:
  Total heap allocated:  741 MiB
  Holding block headers: 16 MiB
  Used small blocks:     0 bytes
  Used ordinary blocks:  554 MiB
  Free small blocks:     5424 bytes
  Free ordinary blocks:  186 MiB
  Ordinary blocks:       344080
  Small blocks:          94
  Holding blocks:        1
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :         13      24                 328
Buffer data                   :          1    4120                4120
Host config                   :          2  (variably sized)        48
Command Tokens                :       3162      72              227728
Command Token Text            :       2362  (variably sized)     77152
Command Token Help            :       2362  (variably sized)     56736
Command Argument              :          2  (variably sized)        48
Command Argument Name         :        534  (variably sized)     12832
FRR POSIX Thread              :         18  (variably sized)      1296
POSIX synchronization primitives:         18  (variably sized)       864
Graph                         :         24       8                 576
Graph Node                    :       3744      32              149984
Hash                          :       1649  (variably sized)     79224
Hash Bucket                   :    1353893      32            54629544
Hash Index                    :        825  (variably sized)  17035136
Hook entry                    :         13      48                 776
Interface                     :          5     248                1240
Connected                     :          9      40                 360
Informational Link Parameters :          2      96                 208
Link List                     :         61      40                2472
Link Node                     :       5076      24              121936
Logging                       :          1      80                  88
Temporary memory              :         19  (variably sized)     28328
Nexthop                       :    1711716     112           205406928
Nexthop label                 :     688148       8            16517520
NetNS Context                 :          2  (variably sized)       128
NetNS Name                    :          1      18                  24
Priority queue                :         10      32                 400
Priority queue data           :         10     256                2640
Prefix                        :         12      48                 672
Privilege information         :          2  (variably sized)        80
Route map                     :          3      64                 216
Route map name                :          3  (variably sized)        72
Route map index               :          4     104                 416
Stream                        :       3444  (variably sized)    624064
Stream FIFO                   :         18      64                1296
Route table                   :         16      48                 896
Route node                    :    1353154  (variably sized) 141309088
Thread                        :       3328     176              612352
Thread master                 :         39  (variably sized)    167336
Thread Poll Info              :         20    8192              164000
Thread stats                  :         37      64                2664
Vector                        :       7543      16              181272
Vector index                  :       7543  (variably sized)    246504
VRF                           :          1     184                 184
VRF bit-map                   :         18       8                 432
VTY                           :          6  (variably sized)     19488
Work queue                    :          3  (variably sized)       360
Work queue item               :          1      24                  24
Work queue name string        :          2  (variably sized)        48
--- qmem Label Manager ---
Label Manager Chunk           :          5      16                 120
--- qmem zebra ---
ZEBRA VRF                     :          1     656                 664
Route Entry                   :     744874      80            65552672
RIB destination               :     740669      48            41481560
RIB table info                :          4      16                  96
Nexthop tracking object       :         10     200                2000
MPLS LSP object               :         22      40                 896
MPLS nexthop object           :         22      56                1232
Zebra Name Space              :          1     312                 312
PTM BFD process registration table.:          3      32                 120
--- qmem Table Manager ---

Memory statistics for ospfd:
System allocator statistics:
  Total heap allocated:  2540 KiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  1560 KiB
  Free small blocks:     1616 bytes
  Free ordinary blocks:  980 KiB
  Ordinary blocks:       102
  Small blocks:          42
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          5      24                 120
Buffer data                   :          1    4120                4120
Host config                   :          2  (variably sized)        48
Command Tokens                :       3383      72              243656
Command Token Text            :       2417  (variably sized)     77144
Command Token Help            :       2417  (variably sized)     58024
Command Argument              :          2  (variably sized)        48
Command Argument Name         :        461  (variably sized)     11080
Graph                         :         16       8                 384
Graph Node                    :       3962      32              158784
Hash                          :        278  (variably sized)     13408
Hash Bucket                   :        656      32               26352
Hash Index                    :        139  (variably sized)     93560
Hook entry                    :          4      48                 240
Interface                     :          5     248                1240
Connected                     :          9      40                 360
Informational Link Parameters :          2      96                 208
Link List                     :         39      40                1560
Link Node                     :        176      24                4304
Logging                       :          1      80                  88
Temporary memory              :          5  (variably sized)       120
Priority queue                :          1      32                  40
Priority queue data           :          1     256                 264
Prefix                        :         12      48                 672
Privilege information         :          2  (variably sized)        96
Route map                     :          3      64                 216
Route map name                :          3  (variably sized)        72
Route map index               :          4     104                 416
Route map rule                :          1      40                  40
Route map rule str            :          1       6                  24
Route map compiled            :          1       4                  24
Stream                        :          3  (variably sized)     98424
Route table                   :         29      48                1640
Route node                    :          2     104                 208
Thread                        :         14     176                2576
Thread master                 :          3  (variably sized)     16712
Thread Poll Info              :          2    8192               16400
Thread stats                  :         30      64                2160
Vector                        :       7963      16              191288
Vector index                  :       7963  (variably sized)    255384
VRF                           :          1     184                 184
VRF bit-map                   :          4       8                  96
VTY                           :          6  (variably sized)     19456
Zclient                       :          1    2984                2984
Redistribution instance IDs   :          3       2                  72
--- qmem ospfd ---
OSPF top                      :          1    3664                3672
OSPF LSDB                     :          1     408                 408
OSPF ext. info                :          2      48                 112
OSPF if info                  :          5      32                 200
OSPF if params                :          7     104                 728
OSPF MPLS parameters          :          5     248                1256
OSPF Extended parameters      :          5     136                 680
OSPF Segment Routing parameters:          4  (variably sized)       416
OSPF opaque function table    :          3     112                 360

Memory statistics for ospf6d:
System allocator statistics:
  Total heap allocated:  1320 KiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  1154 KiB
  Free small blocks:     1504 bytes
  Free ordinary blocks:  166 KiB
  Ordinary blocks:       17
  Small blocks:          41
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          5      24                 120
Buffer data                   :          1    4120                4120
Host config                   :          2  (variably sized)        48
Command Tokens                :       2671      72              192376
Command Token Text            :       1989  (variably sized)     64648
Command Token Help            :       1989  (variably sized)     47752
Command Argument              :          2  (variably sized)        48
Command Argument Name         :        356  (variably sized)      8560
Graph                         :         16       8                 384
Graph Node                    :       3157      32              126568
Hash                          :        218  (variably sized)     10512
Hash Bucket                   :        532      32               21344
Hash Index                    :        109  (variably sized)     31208
Hook entry                    :          2      48                 112
Interface                     :          5     248                1240
Connected                     :          9      40                 360
Link List                     :         18      40                 720
Link Node                     :        132      24                3248
Logging                       :          1      80                  88
Temporary memory              :          5  (variably sized)       120
Priority queue                :          1      32                  40
Priority queue data           :          1     256                 264
Prefix                        :         12      48                 672
Privilege information         :          2  (variably sized)        64
Route map                     :          3      64                 216
Route map name                :          3  (variably sized)        72
Route map index               :          4     104                 416
Route map rule                :          1      40                  40
Route map rule str            :          1       6                  24
Route map compiled            :          1       6                  24
Stream                        :          2   16416               32848
Thread                        :         11     176                2024
Thread master                 :          3  (variably sized)     16712
Thread Poll Info              :          2    8192               16400
Thread stats                  :         10      64                 720
Vector                        :       6354      16              152640
Vector index                  :       6354  (variably sized)    200176
VRF                           :          1     184                 184
VRF bit-map                   :          4       8                  96
VTY                           :          6  (variably sized)     19440
Zclient                       :          1    2984                2984
Redistribution instance IDs   :          3       2                  72
--- qmem ospf6d ---

Memory statistics for ldpd:
System allocator statistics:
  Total heap allocated:  1116 KiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  1077 KiB
  Free small blocks:     1536 bytes
  Free ordinary blocks:  39 KiB
  Ordinary blocks:       3
  Small blocks:          42
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          5      24                 136
Buffer data                   :          1    4120                4120
Host config                   :          2  (variably sized)        48
Command Tokens                :       1737      72              125096
Command Token Text            :       1337  (variably sized)     43496
Command Token Help            :       1337  (variably sized)     32104
Command Argument              :          2  (variably sized)        48
Command Argument Name         :        356  (variably sized)      8544
Graph                         :         18       8                 432
Graph Node                    :       2073      32               83112
Hash                          :        206  (variably sized)      9904
Hash Bucket                   :        373      32               14984
Hash Index                    :        103  (variably sized)     28344
Hook entry                    :          2      48                 112
Interface                     :          4     248                 992
Connected                     :          9      40                 360
Link List                     :         16      40                 640
Link Node                     :        120      24                2912
Logging                       :          1      80                  88
Temporary memory              :          5  (variably sized)       120
Priority queue                :          1      32                  40
Priority queue data           :          1     256                 264
Prefix                        :         12      48                 672
Privilege information         :          2  (variably sized)        64
Stream                        :          2   16416               32848
Thread                        :         11     176                2024
Thread master                 :          3  (variably sized)     16712
Thread Poll Info              :          2    8192               16400
Thread stats                  :         10      64                 720
Vector                        :       4187      16              100600
Vector index                  :       4187  (variably sized)    134024
VRF                           :          1     184                 184
VRF bit-map                   :          3       8                  72
VTY                           :          6  (variably sized)     19440
Zclient                       :          1    2984                2984
Redistribution instance IDs   :          3       2                  72

Memory statistics for bgpd:
System allocator statistics:
  Total heap allocated:  > 2GB
  Holding block headers: 16 MiB
  Used small blocks:     0 bytes
  Used ordinary blocks:  > 2GB
  Free small blocks:     3200 bytes
  Free ordinary blocks:  140 MiB
  Ordinary blocks:       956455
  Small blocks:          70
  Holding blocks:        1
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          6      24                 144
Buffer data                   :          1  (variably sized)      4120
Host config                   :          2  (variably sized)        48
Command Tokens                :       9807      72              706568
Command Token Text            :       7240  (variably sized)    248144
Command Token Help            :       7240  (variably sized)    174320
Command Argument              :          2  (variably sized)        48
Command Argument Name         :       1530  (variably sized)     36832
FRR POSIX Thread              :          4  (variably sized)       288
POSIX synchronization primitives:          4  (variably sized)       192
Graph                         :         35       8                 840
Graph Node                    :      11657      32              467496
Hash                          :        708  (variably sized)     34048
Hash Bucket                   :    2761302      32           111714896
Hash Index                    :        355  (variably sized)  33137728
Hook entry                    :          2      48                 112
Interface                     :          4     248                 992
Connected                     :          9      40                 376
Informational Link Parameters :          2      96                 208
Link List                     :         42      40                1696
Link Node                     :        403      24                9720
Logging                       :          1      80                  88
Temporary memory              :         10  (variably sized)       288
Nexthop                       :         12     112                1440
Nexthop label                 :          5       8                 120
Priority queue                :          3      32                 120
Priority queue data           :          3     256                 792
Prefix                        :         12      48                 704
Privilege information         :          2  (variably sized)        96
Ring buffer                   :         12  (variably sized)    246048
Route map                     :          3      64                 216
Route map name                :          5  (variably sized)       120
Route map index               :          4     104                 416
Route map rule                :          6      40                 240
Route map rule str            :          6  (variably sized)       144
Route map compiled            :          8  (variably sized)       192
Route map dependency          :          1      24                  24
Skip List                     :          2      56                 144
Skip Node                     :          4     160                 672
Socket union                  :         13      28                 536
Stream                        :    1295083  (variably sized) 264771096
Stream FIFO                   :         12      64                 912
Route table                   :        106      48                5936
Thread                        :         32     176                5920
Thread master                 :         11  (variably sized)     50184
Thread Poll Info              :          6    8192               49200
Thread stats                  :         25      64                1800
Vector                        :      23391      16              562152
Vector index                  :      23391  (variably sized)    741432
VRF                           :          1     184                 184
VRF bit-map                   :          3       8                  72
VTY                           :          6  (variably sized)     19440
Work queue                    :          7     144                1064
Work queue item               :     692394      24            26034208
Work queue name string        :          7  (variably sized)       168
Zclient                       :          2    2984                5968
Redistribution instance IDs   :          6       2                 144
--- qmem rfapi ---
NVE Configuration             :          1    2648                2648
RFAPI Generic                 :          1     296                 296
RFAPI Import Table            :          1     208                 216
--- qmem bgpd ---
BGP Label FIFO                :          1      48                  56
BGP PBR Context               :          1      16                  24
BGP instance                  :          2  (variably sized)      4832
BGP listen socket details     :          2      48                 112
BGP peer                      :          9   21008              189144
BGP peer hostname             :         13  (variably sized)       312
Peer group                    :          1      64                  72
BGP Peer group hostname       :          1       4                  24
BGP peer af                   :          4      80                 352
BGP update group              :          3     104                 312
BGP update subgroup           :          3     240                 744
BGP packet                    :          3      56                 168
BGP attribute                 :     637254     232           147853472
BGP aspath                    :     177591      40             7105592
BGP aspath seg                :     177869      24             4268856
BGP aspath segment data       :     177869  (variably sized)   4812872
BGP aspath str                :     177591  (variably sized)  11850936
BGP table                     :         93      40                3736
BGP node                      :    1367085     160           230107064
BGP route                     :    2910853     112           349497240
BGP ancillary route info      :    2195888     216           476453040
BGP connected                 :          6       4                 144
BGP adv attr                  :     553928      24            13294272
BGP adv                       :    3334882      72           240361440
BGP synchronise               :        129      72                9496
BGP adj out                   :    3398427      72           244833880
BGP multipath info            :     373007      40            15796824
community                     :      22752      40              918800
community val                 :      22752  (variably sized)   1249872
community str                 :      22751  (variably sized)   3353336
extcommunity                  :        551      32               22776
extcommunity val              :        551  (variably sized)     16616
extcommunity str              :        551  (variably sized)     65624
community-list                :          2      56                 112
community-list name           :          2  (variably sized)        48
community-list entry          :          3      48                 168
community-list handler        :          1      96                 104
Cluster list                  :        102      24                2448
Cluster list val              :        102  (variably sized)      3232
BGP Process queue             :         65      32                2936
BGP node clear queue          :     692329       8            16945832
BGP transit attr              :          5      24                 120
BGP transit val               :          5  (variably sized)       120
BGP nexthop                   :         10      72                 752
BGP own address               :          4       8                  96
BGP Filter Information        :         13  (variably sized)       312
Large Community               :         58      40                2336
Large Community display string:         58  (variably sized)      2352
Large Community value         :         58  (variably sized)      1696

Memory statistics for isisd:
System allocator statistics:
  Total heap allocated:  1328 KiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  1215 KiB
  Free small blocks:     1808 bytes
  Free ordinary blocks:  113 KiB
  Ordinary blocks:       7
  Small blocks:          48
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          5      24                 120
Buffer data                   :          1    4120                4120
Host config                   :          2  (variably sized)        48
Command Tokens                :       2176      72              156720
Command Token Text            :       1708  (variably sized)     55296
Command Token Help            :       1708  (variably sized)     41008
Command Argument              :          2  (variably sized)        48
Command Argument Name         :        285  (variably sized)      6856
Graph                         :         16       8                 384
Graph Node                    :       2650      32              106080
Hash                          :        282  (variably sized)     13648
Hash Bucket                   :        710      32               28624
Hash Index                    :        142  (variably sized)     99312
Hook entry                    :          4      48                 224
Interface                     :          5     248                1240
Connected                     :          9      40                 360
Informational Link Parameters :          2      96                 208
Link List                     :        278      40               11136
Link Node                     :        456      24               11120
Logging                       :          1      80                  88
Temporary memory              :          9  (variably sized)      4296
Priority queue                :          1      32                  40
Priority queue data           :          1     256                 264
Prefix                        :         21      48                1176
Privilege information         :          2  (variably sized)        64
Route map                     :          3      64                 216
Route map name                :          3  (variably sized)        72
Route map index               :          4     104                 416
Route map rule                :          1      40                  40
Route map rule str            :          1       6                  24
Route map compiled            :          1       4                  24
Skip List                     :          9      56                 504
Skip Node                     :         18  (variably sized)      3024
Stream                        :         29  (variably sized)     49432
Route table                   :         13      48                 744
Route node                    :         87     112               10488
Thread                        :         31     176                5736
Thread master                 :          3  (variably sized)     16712
Thread Poll Info              :          2    8192               16400
Thread stats                  :         25      64                1816
Vector                        :       5339      16              128280
Vector index                  :       5339  (variably sized)    169608
VRF                           :          1     184                 184
VRF bit-map                   :          5       8                 120
VTY                           :          6  (variably sized)     19440
Zclient                       :          1    2984                2984
Redistribution instance IDs   :          3       2                  72
--- qmem isisd ---
ISIS LSP Hash                 :          3       8                  72
ISIS                          :          1     160                 168
ISIS circuit                  :          4     704                2848
ISIS LSP                      :         23     344                7912
ISIS adjacency                :          3     440                1336
ISIS adjacency info           :          9  (variably sized)       216
ISIS area                     :          2    3488                6992
ISIS area address             :          1      21                  24
ISIS dyn hostname             :          5     280                1416
ISIS SPFtree                  :          9     112                1080
ISIS vertex                   :         78     112                9360
ISIS route info               :         35      32                1400
ISIS nexthop                  :          2      16                  48
ISIS nexthop6                 :          8      40                 320
ISIS dictionary               :          3     104                 312
ISIS dictionary node          :         23      48                1320
ISIS redistributed route info :         12      12                 288
ISIS MPLS_TE parameters       :          4     144                 608
ISIS MT Area Setting          :          2       4                  48
ISIS MT Circuit Setting       :          2       4                  48
ISIS MT Adjacency Info        :          3       2                  72
ISIS TLVs                     :        194  (variably sized)     25088

Memory statistics for watchfrr:
System allocator statistics:
  Total heap allocated:  264 KiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  218 KiB
  Free small blocks:     1248 bytes
  Free ordinary blocks:  46 KiB
  Ordinary blocks:       1
  Small blocks:          35
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          2      24                  48
Buffer data                   :          1    4120                4120
Host config                   :          1      12                  24
Command Tokens                :        365      72               26280
Command Token Text            :        299  (variably sized)      9832
Command Token Help            :        299  (variably sized)      7176
Command Argument              :          2  (variably sized)        48
Command Argument Name         :         38  (variably sized)       912
Graph                         :          6       8                 144
Graph Node                    :        462      32               18480
Hash                          :         18  (variably sized)       864
Hash Bucket                   :        121      32                4840
Hash Index                    :          9  (variably sized)      3848
Hook entry                    :          2      48                 112
Link List                     :          5      40                 200
Link Node                     :         12      24                 288
Logging                       :          1      80                  88
Temporary memory              :          5  (variably sized)       120
Priority queue                :          1      32                  40
Priority queue data           :          1     256                 264
Thread                        :         18     176                3312
Thread master                 :          3  (variably sized)     16712
Thread Poll Info              :          2    8192               16400
Thread stats                  :          7      64                 520
Vector                        :        941      16               22584
Vector index                  :        941  (variably sized)     30600
VTY                           :          3  (variably sized)      9720

Memory statistics for staticd:
System allocator statistics:
  Total heap allocated:  660 KiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  587 KiB
  Free small blocks:     1424 bytes
  Free ordinary blocks:  73 KiB
  Ordinary blocks:       1
  Small blocks:          40
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          3      24                  72
Buffer data                   :          1    4120                4120
Host config                   :          2  (variably sized)        48
Command Tokens                :       1137      72               81880
Command Token Text            :        896  (variably sized)     28496
Command Token Help            :        896  (variably sized)     21520
Command Argument              :          2  (variably sized)        48
Command Argument Name         :        211  (variably sized)      5064
Graph                         :         11       8                 264
Graph Node                    :       1339      32               53720
Hash                          :        202  (variably sized)      9712
Hash Bucket                   :        238      32                9536
Hash Index                    :        101  (variably sized)     37352
Hook entry                    :          2      48                 112
Interface                     :          4     248                 992
Connected                     :          9      40                 360
Link List                     :         17      40                 680
Link Node                     :        117      24                2888
Logging                       :          1      80                  88
Temporary memory              :          7  (variably sized)       408
Priority queue                :          1      32                  40
Priority queue data           :          1     256                 264
Prefix                        :         13      48                 728
Stream                        :          2   16416               32848
Route table                   :          4      48                 256
Route node                    :          5     104                 520
Thread                        :          5     176                 920
Thread master                 :          3  (variably sized)     16712
Thread Poll Info              :          2    8192               16400
Thread stats                  :          6      64                 432
Vector                        :       2705      16               64984
Vector index                  :       2705  (variably sized)     87368
VRF                           :          1     184                 184
VRF bit-map                   :          3       8                  72
VTY                           :          3  (variably sized)      9720
Zclient                       :          1    2984                2984
Redistribution instance IDs   :          3       2                  72
--- qmem staticd ---
Static Route                  :          3     192                 600

Memory statistics for bfdd:
System allocator statistics:
  Total heap allocated:  528 KiB
  Holding block headers: 0 bytes
  Used small blocks:     0 bytes
  Used ordinary blocks:  458 KiB
  Free small blocks:     1360 bytes
  Free ordinary blocks:  70 KiB
  Ordinary blocks:       2
  Small blocks:          37
  Holding blocks:        0
(see system documentation for 'mallinfo' for meaning)
--- qmem libfrr ---
Buffer                        :          5      24                 120
Buffer data                   :          1    4120                4120
Host config                   :          2  (variably sized)        48
Command Tokens                :        778      72               56016
Command Token Text            :        617  (variably sized)     19352
Command Token Help            :        617  (variably sized)     14824
Command Argument              :          2  (variably sized)        48
Command Argument Name         :         84  (variably sized)      2016
Graph                         :          9       8                 232
Graph Node                    :        936      32               37456
Hash                          :        198  (variably sized)      9504
Hash Bucket                   :        183      32                7320
Hash Index                    :         99  (variably sized)     36568
Hook entry                    :          2      48                 112
Link List                     :          8      40                 320
Link Node                     :        105      24                2536
Logging                       :          2      80                 176
Temporary memory              :          5  (variably sized)       120
Priority queue                :          1      32                  40
Priority queue data           :          1     256                 264
Privilege information         :          2  (variably sized)        48
Stream                        :          2   16416               32848
Thread                        :         12     176                2208
Thread master                 :          3  (variably sized)     16712
Thread Poll Info              :          2    8192               16400
Thread stats                  :          8      64                 576
Vector                        :       1895      16               45528
Vector index                  :       1895  (variably sized)     62584
VRF                           :          1     184                 184
VTY                           :          6  (variably sized)     19440
Zclient                       :          1    2984                2984
Redistribution instance IDs   :          3       2                  72
--- qmem Bidirectional Forwarding Detection Daemon ---
long-lived control socket memory:          3      40                 120

I don't have a great understanding of frr, as I never contributed as developer, so I'm kindly asking - does any of you have an inclination based on the above what could be memory leaking here?

donaldsharp commented 6 years ago

BGP's usage of streams is very large:

Stream : 1295083 (variably sized) 264771096

And I see a bunch of data associated with a clear * command. What are you doing on this machine? Maybe we have a bug somewhere in this area.

ghost commented 6 years ago

@donaldsharp I already found that very weird too. I'm pretty much getting 3 full tables of 3 different routers, then I'm sending back the routes again (so pretty much, I have this machine as route reflector in Frankfurt, and my backbone routers based on Juniper are meshed to that machine).

The data associated with the clear command could be because the machine is sitting at this for a while:

10.42.42.16     4      49004  943243  958567        0    0    0 00:16:07     Clearing

there's genuinely no reason why that session would flap at all, though, I do believe that it terminated the session because the entire thing is close to running out of memory.

donaldsharp commented 6 years ago

If we were running out of memory bgp would just be killed and we would restart. There is nothing more complicated here.

Let's get some data gathered from perf about bgp:

https://github.com/FRRouting/frr/wiki/Perf-Recording

Can we gather a flame graph for bgp? I am not sure what is going on here and would like to see some data gathered. Can we get some idea of how you are configuring your peers?

ghost commented 6 years ago

I'll have a look into PERF recording in a second. The peers are set up completely trivial:

!
router bgp 49004
 coalesce-time 20000
 neighbor igp peer-group
 neighbor igp remote-as 49004
 neighbor igp update-source 10.42.42.40
 neighbor 10.42.42.9 peer-group igp
 neighbor 10.42.42.16 peer-group igp
 neighbor 10.42.42.34 peer-group igp
 !
 address-family ipv4 unicast
  neighbor igp route-reflector-client
  neighbor igp route-map igp-block out
  neighbor 10.42.42.34 addpath-tx-all-paths
 exit-address-family
 !
 address-family ipv6 unicast
  neighbor igp route-reflector-client
  neighbor igp route-map igp-block out
 exit-address-family
!

the addpath-tx-all-paths is in there for "weird" internal reasons, that being pretty much that this peer needs to receive all available routes, not just preferred, due to special network preferences happening on that router. Shouldn't be of relevance though I do believe.

[Edit]: the igp-block route-map literally does nothing at the moment:

route-map igp-block permit 1
!
ghost commented 6 years ago

I've tried taking the flame graph, I hope I didn't mess it up, never used before. Please do let me know if it's useless. https://ufile.io/9bywe

ghost commented 6 years ago

Here's the same taken from the zebra pid, not sure though if it helps as it literally shows bgpd and almost the same as the other flame graph. https://ufile.io/7cb6d

donaldsharp commented 6 years ago

let's just attach the flamegraphs to this issue? We'll loose them otherwise right?

BGP is taking so long because it is swapping heavily.

ghost commented 6 years ago

@donaldsharp It won't let me attach an .svg right here where I reply. Do you see any alternative way? Either way, I entirely understand it's taking ages for sessions to clear and converge, but my point merely is that the box has 2GB of RAM, of which 1.9GB are used by FRR just to handle 3 BGP full feeds, which seems to be a bit too much, making me suspect a memory leak.

ghost commented 6 years ago

@donaldsharp I've tried to enable bgpd debugging, perhaps there's anything obvious that would explode the memory. I've seen large amounts of that:

Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.122.0/24 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.120.0/23 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.116.0/22 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.115.0/24 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.114.0/24 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.112.0/23 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.108.0/22 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.106.0/23 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.100.0/23 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.96.0/22 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.94.0/23 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.93.0/24 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.90.0/24 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.88.0/23 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.84.0/22 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.82.0/23 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.81.0/24 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.80.0/24 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.76.0/22 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.74.0/23 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.68.0/23 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.66.0/23 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.65.0/24 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.64.0/24 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.62.0/23 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.61.0/24 IPv4 unicast -- DENIED due to: reflected from the same cluster;
Nov  4 16:43:19 bb1 bgpd[2221]: 10.42.42.9 rcvd UPDATE about 147.194.60.0/24 IPv4 unicast -- DENIED due to: reflected from the same cluster;

and that:

Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 206.201.74.0/24 with addpath ID 10966111 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 206.201.77.0/24 with addpath ID 10966110 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 206.201.79.0/24 with addpath ID 10966109 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 206.201.80.0/21 with addpath ID 11263788 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 206.201.88.0/24 with addpath ID 11347202 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 206.201.89.0/24 with addpath ID 11347203 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 206.201.92.0/24 with addpath ID 11347204 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 206.201.93.0/24 with addpath ID 11347205 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 104.153.189.0/24 with addpath ID 12812985 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 104.193.184.0/24 with addpath ID 12812814 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 104.193.187.0/24 with addpath ID 12812812 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 191.96.36.0/24 with addpath ID 12812808 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 191.96.16.0/24 with addpath ID 12812809 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 179.61.219.0/24 with addpath ID 12812810 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 173.239.198.0/24 with addpath ID 12812811 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 104.193.188.0/24 with addpath ID 12812807 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 104.193.189.0/24 with addpath ID 12812806 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 104.193.201.0/24 with addpath ID 12812805 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 104.193.216.0/21 with addpath ID 12812804 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 104.193.222.0/24 with addpath ID 12812803 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 104.194.4.0/24 with addpath ID 12812802 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 116.93.128.0/24 with addpath ID 12812801 IPv4 unicast -- unreachable
Nov  4 16:43:19 bb1 bgpd[2221]: u12:s15 send UPDATE 104.194.16.0/22 with addpath ID 12812800 IPv4 unicast -- unreachable

and

Nov  4 16:43:18 bb1 bgpd[2221]: subgroup_process_announce_selected: p=199.202.145.0/24, selected=0x55db48a4fb90
Nov  4 16:43:18 bb1 bgpd[2221]: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=104.194.31.0/24
Nov  4 16:43:18 bb1 bgpd[2221]: subgroup_process_announce_selected: p=104.194.31.0/24, selected=(nil)
Nov  4 16:43:18 bb1 bgpd[2221]: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=104.194.31.0/24
Nov  4 16:43:18 bb1 bgpd[2221]: subgroup_process_announce_selected: p=104.194.31.0/24, selected=0x55db145ed900
Nov  4 16:43:18 bb1 bgpd[2221]: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=104.194.30.0/24
Nov  4 16:43:18 bb1 bgpd[2221]: subgroup_process_announce_selected: p=104.194.30.0/24, selected=(nil)
Nov  4 16:43:18 bb1 bgpd[2221]: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=104.194.30.0/24
Nov  4 16:43:18 bb1 bgpd[2221]: subgroup_process_announce_selected: p=104.194.30.0/24, selected=0x55db145e7870
Nov  4 16:43:18 bb1 bgpd[2221]: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=104.194.29.0/24
Nov  4 16:43:18 bb1 bgpd[2221]: subgroup_process_announce_selected: p=104.194.29.0/24, selected=(nil)
Nov  4 16:43:18 bb1 bgpd[2221]: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=104.194.29.0/24
Nov  4 16:43:18 bb1 bgpd[2221]: subgroup_process_announce_selected: p=104.194.29.0/24, selected=0x55db145e15b0
Nov  4 16:43:18 bb1 bgpd[2221]: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=104.194.28.0/24
Nov  4 16:43:18 bb1 bgpd[2221]: subgroup_process_announce_selected: p=104.194.28.0/24, selected=(nil)
Nov  4 16:43:18 bb1 bgpd[2221]: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=104.194.28.0/24
Nov  4 16:43:18 bb1 bgpd[2221]: subgroup_process_announce_selected: p=104.194.28.0/24, selected=0x55db145dfde0
Nov  4 16:43:18 bb1 bgpd[2221]: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=104.194.28.0/22
Nov  4 16:43:18 bb1 bgpd[2221]: subgroup_process_announce_selected: p=104.194.28.0/22, selected=(nil)
Nov  4 16:43:18 bb1 bgpd[2221]: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=104.194.28.0/22
Nov  4 16:43:18 bb1 bgpd[2221]: subgroup_process_announce_selected: p=104.194.28.0/22, selected=0x55db5d02c630
Nov  4 16:43:18 bb1 bgpd[2221]: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=104.194.19.0/24
Nov  4 16:43:18 bb1 bgpd[2221]: subgroup_process_announce_selected: p=104.194.19.0/24, selected=(nil)
Nov  4 16:43:18 bb1 bgpd[2221]: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=104.194.19.0/24

the latter is pretty much creating hundreds of lines per second. If that's weird, what would cause it?

liuxyon commented 4 years ago

I also encountered a very large memory footprint, when i get an v4 full route from upstream, then frr Consumes 1.5G of memory. Really need to solve the problem of large memory consumption

i have config is bgp bestpath as-path confed bgp bestpath med confed

neighbor 10.88.45.149 remote-as 108 neighbor 10.88.45.149 enforce-first-as neighbor 10.88.45.149 update-source 10.88.45.150 neighbor 10.88.45.149 activate neighbor 10.88.45.149 next-hop-self neighbor 10.88.45.149 remove-private-AS all neighbor 10.88.45.149 prefix-list ipv4in in neighbor 10.88.45.149 prefix-list myout out

ton31337 commented 2 years ago

How about the latest (master) version?