search

els0r / goProbe

High-performance IP packet metadata aggregation and efficient storage and querying of flows
GNU General Public License v2.0
12 stars 4 forks source link

[feature] Keep Alive for goQuery: log progress during query execution #319

Closed els0r closed 5 months ago

els0r commented 5 months ago

This commit provides a simple (!) keep-alive mechanism, which can be used to inform the caller of goQuery that a query is still running.

The --query.keepalive or -k shorthand specifies an interval after which the caller gets a query processing update. By default, it is 0, which equates no keep alive.

Example Output

ts=2024-05-23T14:39:01.715Z level=info msg="processed work dir" version=devel worker.id=7 iface=eth0 dir=.db/eth0/2024/03/1710460800 stats.bytes_loaded=0 stats.bytes_decompressed=3848214 stats.blocks_processed=17860 stats.blocks_corrupted=0 stats.directories_processed=31 stats.workloads=1
ts=2024-05-23T14:39:01.931Z level=info msg="processed workload" version=devel iface=eth0 dir=.db/eth0 stats.bytes_loaded=0 stats.bytes_decompressed=7357134 stats.blocks_processed=68980 stats.blocks_corrupted=0 stats.directories_processed=60 stats.workloads=2
ts=2024-05-23T14:39:01.933Z level=info msg="processed workload" version=devel iface=eth0 dir=.db/eth0 stats.bytes_loaded=0 stats.bytes_decompressed=10069197 stats.blocks_processed=105844 stats.blocks_corrupted=0 stats.directories_processed=92 stats.workloads=3
ts=2024-05-23T14:39:01.933Z level=info msg="processed workload" version=devel iface=eth0 dir=.db/eth0 stats.bytes_loaded=0 stats.bytes_decompressed=12768653 stats.blocks_processed=142716 stats.blocks_corrupted=0 stats.directories_processed=124 stats.workloads=4
ts=2024-05-23T14:39:01.968Z level=info msg="processed workload" version=devel iface=eth0 dir=.db/eth0 stats.bytes_loaded=0 stats.bytes_decompressed=16810721 stats.blocks_processed=179588 stats.blocks_corrupted=0 stats.directories_processed=156 stats.workloads=5
ts=2024-05-23T14:39:01.969Z level=info msg="processed workload" version=devel iface=eth0 dir=.db/eth0 stats.bytes_loaded=0 stats.bytes_decompressed=19428092 stats.blocks_processed=216456 stats.blocks_corrupted=0 stats.directories_processed=188 stats.workloads=6
ts=2024-05-23T14:39:01.970Z level=info msg="processed workload" version=devel iface=eth0 dir=.db/eth0 stats.bytes_loaded=0 stats.bytes_decompressed=22129050 stats.blocks_processed=253324 stats.blocks_corrupted=0 stats.directories_processed=220 stats.workloads=7

                           packets   packets            bytes     bytes       
         sip         dip        in       out      %        in       out      %
  200.13.1.2  200.24.1.2   41.38 M   49.49 M  25.26   5.09 GB   6.09 GB  22.47
                               ...       ...              ...       ...       
                          191.16 M  168.53 M         28.11 GB  21.62 GB       

     Totals:                        359.69 M                   49.73 GB  

Timespan    : [2023-09-06 23:58:54, 2024-04-13 20:35:00] (219d20h36m0s)
Interface   : eth0
Sorted by   : accumulated data volume (sent and received)
Query stats : displayed top 1 hits out of 506 in 7.448s

Other Changes

The stats label in the log output stems from a WorkloadStats struct which tracks interactions with the DB engine. Mainly how much data is loaded and processed.

It is meant to be extended in future iterations.