search

netenglabs / suzieq

Using network observability to operate and design healthier networks
https://www.stardustsystems.net/
Apache License 2.0
792 stars 106 forks source link

coalescer has created very large directories with a null namespace that has files that are of increasing size #324

Closed jopietsch closed 3 years ago

jopietsch commented 3 years ago

Describe the bug in both ifCounters and ospfIf, there is a blank namespace. Inside those two blank namespaces, each parquet file is larger then the last. So the parquet file written at 17:36 is bigger than the file written at 16:35, and so on. Each new file is bigger than the previous, as if it is one table that keeps growing.

(suzieq-QZqWZG2D-py3.8) jpietsch@jpietsch-server:~/suzieq$ ls -l ~/parquet-out/coalesced/ospfIf/sqvers\=1.0/namespace\=/ |head
total 13720
-rw-rw-r-- 1 jpietsch jpietsch  16776 Feb 17 09:45 sqc-h-1613577600-1613581200.parquet
-rw-rw-r-- 1 jpietsch jpietsch  17415 Feb 18 09:11 sqc-h-1613581200-1613584800.parquet
-rw-rw-r-- 1 jpietsch jpietsch  17415 Feb 18 09:11 sqc-h-1613584800-1613588400.parquet
-rw-rw-r-- 1 jpietsch jpietsch  17845 Feb 18 10:11 sqc-h-1613667600-1613671200.parquet
-rw-rw-r-- 1 jpietsch jpietsch  18221 Feb 18 11:12 sqc-h-1613671200-1613674800.parquet
-rw-rw-r-- 1 jpietsch jpietsch  18709 Feb 18 12:12 sqc-h-1613674800-1613678400.parquet
-rw-rw-r-- 1 jpietsch jpietsch  19136 Feb 18 13:13 sqc-h-1613678400-1613682000.parquet
-rw-rw-r-- 1 jpietsch jpietsch  19636 Feb 18 14:13 sqc-h-1613682000-1613685600.parquet
-rw-rw-r-- 1 jpietsch jpietsch  20024 Feb 18 15:14 sqc-h-1613685600-1613689200.parquet

In ifCounters it starts at 13778 bytes, and keeps growing, and then resets back to 13778

(suzieq-QZqWZG2D-py3.8) jpietsch@jpietsch-server:~/suzieq$ ls -l ~/parquet-out/coalesced/ifCounters/sqvers\=1.0/namespace\=/ |head -50
total 4323840
-rw-rw-r-- 1 jpietsch jpietsch   706516 Feb 17 09:45 sqc-h-1613577600-1613581200.parquet
-rw-rw-r-- 1 jpietsch jpietsch  1347205 Feb 18 09:11 sqc-h-1613581200-1613584800.parquet
-rw-rw-r-- 1 jpietsch jpietsch    13778 Feb 18 09:11 sqc-h-1613584800-1613588400.parquet
-rw-rw-r-- 1 jpietsch jpietsch    62287 Feb 18 10:11 sqc-h-1613667600-1613671200.parquet
-rw-rw-r-- 1 jpietsch jpietsch   615696 Feb 18 11:12 sqc-h-1613671200-1613674800.parquet
-rw-rw-r-- 1 jpietsch jpietsch  1256607 Feb 18 12:12 sqc-h-1613674800-1613678400.parquet
-rw-rw-r-- 1 jpietsch jpietsch  1922094 Feb 18 13:13 sqc-h-1613678400-1613682000.parquet
-rw-rw-r-- 1 jpietsch jpietsch  2600163 Feb 18 14:14 sqc-h-1613682000-1613685600.parquet
-rw-rw-r-- 1 jpietsch jpietsch  3227511 Feb 18 15:14 sqc-h-1613685600-1613689200.parquet
-rw-rw-r-- 1 jpietsch jpietsch  3927720 Feb 18 16:15 sqc-h-1613689200-1613692800.parquet
-rw-rw-r-- 1 jpietsch jpietsch  4573525 Feb 18 17:15 sqc-h-1613692800-1613696400.parquet
-rw-rw-r-- 1 jpietsch jpietsch  5251116 Feb 18 18:16 sqc-h-1613696400-1613700000.parquet
-rw-rw-r-- 1 jpietsch jpietsch  5864474 Feb 18 19:17 sqc-h-1613700000-1613703600.parquet
-rw-rw-r-- 1 jpietsch jpietsch  6559393 Feb 18 20:17 sqc-h-1613703600-1613707200.parquet
-rw-rw-r-- 1 jpietsch jpietsch  7211885 Feb 18 21:18 sqc-h-1613707200-1613710800.parquet
-rw-rw-r-- 1 jpietsch jpietsch  7851287 Feb 18 22:18 sqc-h-1613710800-1613714400.parquet
-rw-rw-r-- 1 jpietsch jpietsch  8555540 Feb 18 23:19 sqc-h-1613714400-1613718000.parquet
-rw-rw-r-- 1 jpietsch jpietsch  9179674 Feb 19 00:20 sqc-h-1613718000-1613721600.parquet
-rw-rw-r-- 1 jpietsch jpietsch  9891155 Feb 19 01:20 sqc-h-1613721600-1613725200.parquet
-rw-rw-r-- 1 jpietsch jpietsch 10525191 Feb 19 02:21 sqc-h-1613725200-1613728800.parquet
-rw-rw-r-- 1 jpietsch jpietsch 11242067 Feb 19 03:22 sqc-h-1613728800-1613732400.parquet
-rw-rw-r-- 1 jpietsch jpietsch 11885702 Feb 19 04:22 sqc-h-1613732400-1613736000.parquet
-rw-rw-r-- 1 jpietsch jpietsch 12545695 Feb 19 05:23 sqc-h-1613736000-1613739600.parquet
-rw-rw-r-- 1 jpietsch jpietsch 13203199 Feb 19 06:24 sqc-h-1613739600-1613743200.parquet
-rw-rw-r-- 1 jpietsch jpietsch 13801704 Feb 19 07:24 sqc-h-1613743200-1613746800.parquet
-rw-rw-r-- 1 jpietsch jpietsch 14521663 Feb 19 08:25 sqc-h-1613746800-1613750400.parquet
-rw-rw-r-- 1 jpietsch jpietsch 15194054 Feb 19 09:26 sqc-h-1613750400-1613754000.parquet
-rw-rw-r-- 1 jpietsch jpietsch 15851845 Feb 19 10:26 sqc-h-1613754000-1613757600.parquet
-rw-rw-r-- 1 jpietsch jpietsch 16497897 Feb 19 11:27 sqc-h-1613757600-1613761200.parquet
-rw-rw-r-- 1 jpietsch jpietsch 17165086 Feb 19 12:28 sqc-h-1613761200-1613764800.parquet
-rw-rw-r-- 1 jpietsch jpietsch 17821136 Feb 19 13:29 sqc-h-1613764800-1613768400.parquet
-rw-rw-r-- 1 jpietsch jpietsch 18475759 Feb 22 11:26 sqc-h-1613768400-1613772000.parquet
-rw-rw-r-- 1 jpietsch jpietsch    13778 Feb 22 11:26 sqc-h-1613772000-1613775600.parquet
-rw-rw-r-- 1 jpietsch jpietsch    98002 Feb 22 12:27 sqc-h-1614020400-1614024000.parquet
-rw-rw-r-- 1 jpietsch jpietsch   480155 Feb 22 13:27 sqc-h-1614024000-1614027600.parquet
-rw-rw-r-- 1 jpietsch jpietsch  1129679 Feb 22 14:28 sqc-h-1614027600-1614031200.parquet
-rw-rw-r-- 1 jpietsch jpietsch  1785962 Feb 22 15:29 sqc-h-1614031200-1614034800.parquet
-rw-rw-r-- 1 jpietsch jpietsch  2460710 Feb 22 16:29 sqc-h-1614034800-1614038400.parquet
-rw-rw-r-- 1 jpietsch jpietsch  3089259 Feb 22 17:30 sqc-h-1614038400-1614042000.parquet
-rw-rw-r-- 1 jpietsch jpietsch  3745545 Feb 22 18:31 sqc-h-1614042000-1614045600.parquet
-rw-rw-r-- 1 jpietsch jpietsch  4400993 Feb 22 19:31 sqc-h-1614045600-1614049200.parquet
-rw-rw-r-- 1 jpietsch jpietsch  5085177 Feb 22 20:32 sqc-h-1614049200-1614052800.parquet
-rw-rw-r-- 1 jpietsch jpietsch  5749970 Feb 22 21:33 sqc-h-1614052800-1614056400.parquet
-rw-rw-r-- 1 jpietsch jpietsch  6369976 Feb 22 22:33 sqc-h-1614056400-1614060000.parquet
-rw-rw-r-- 1 jpietsch jpietsch  7064344 Feb 22 23:34 sqc-h-1614060000-1614063600.parquet
-rw-rw-r-- 1 jpietsch jpietsch  7735054 Feb 23 00:35 sqc-h-1614063600-1614067200.parquet
-rw-rw-r-- 1 jpietsch jpietsch  8378625 Feb 23 01:35 sqc-h-1614067200-1614070800.parquet
-rw-rw-r-- 1 jpietsch jpietsch  9069452 Feb 23 02:36 sqc-h-1614070800-1614074400.parquet
-rw-rw-r-- 1 jpietsch jpietsch  9720162 Feb 23 03:37 sqc-h-1614074400-1614078000.parquet

I noticed because my coalesced directory is much bigger than my _archived directory

(suzieq-QZqWZG2D-py3.8) jpietsch@jpietsch-server:~/suzieq$ du -hs ~/parquet-out/*
1.1G    /home/jpietsch/parquet-out/_archived
508K    /home/jpietsch/parquet-out/arpnd
320K    /home/jpietsch/parquet-out/bgp
4.9G    /home/jpietsch/parquet-out/coalesced
572K    /home/jpietsch/parquet-out/device
152K    /home/jpietsch/parquet-out/evpnVni
508K    /home/jpietsch/parquet-out/fs
70M     /home/jpietsch/parquet-out/ifCounters
508K    /home/jpietsch/parquet-out/interfaces
352K    /home/jpietsch/parquet-out/lldp
332K    /home/jpietsch/parquet-out/macs
108K    /home/jpietsch/parquet-out/mlag
268K    /home/jpietsch/parquet-out/ospfIf
268K    /home/jpietsch/parquet-out/ospfNbr
508K    /home/jpietsch/parquet-out/routes
45M     /home/jpietsch/parquet-out/sqPoller
460K    /home/jpietsch/parquet-out/time
18M     /home/jpietsch/parquet-out/topcpu
18M     /home/jpietsch/parquet-out/topmem
236K    /home/jpietsch/parquet-out/vlan
jopietsch commented 3 years ago

inside coalesced/ifCounters, the blank namespace is much bigger than the others.

(suzieq-QZqWZG2D-py3.8) jpietsch@jpietsch-server:~/parquet-out/coalesced/ifCounters/sqvers=1.0/namespace=$ du -hs ~/parquet-out/coalesced/ifCounters/sqvers\=1.0/*
4.2G    /home/jpietsch/parquet-out/coalesced/ifCounters/sqvers=1.0/namespace=
7.1M    /home/jpietsch/parquet-out/coalesced/ifCounters/sqvers=1.0/namespace=dual-bgp
202M    /home/jpietsch/parquet-out/coalesced/ifCounters/sqvers=1.0/namespace=dual-evpn
5.6M    /home/jpietsch/parquet-out/coalesced/ifCounters/sqvers=1.0/namespace=eos
5.7M    /home/jpietsch/parquet-out/coalesced/ifCounters/sqvers=1.0/namespace=junos
5.6M    /home/jpietsch/parquet-out/coalesced/ifCounters/sqvers=1.0/namespace=nxos
7.0M    /home/jpietsch/parquet-out/coalesced/ifCounters/sqvers=1.0/namespace=ospf-ibgp
5.7M    /home/jpietsch/parquet-out/coalesced/ifCounters/sqvers=1.0/namespace=ospf-single
20M     /home/jpietsch/parquet-out/coalesced/ifCounters/sqvers=1.0/namespace=single
9.3M    /home/jpietsch/parquet-out/coalesced/ifCounters/sqvers=1.0/namespace=single-evpn
ddutt commented 3 years ago

Fixed via commit 32265e8cf757a3bcaf90b39cc99dbd97321dbe16 and 528168648de6e1a77017085755f2605ad6078c90