mosajjal / dnsmonster

Passive DNS Capture and Monitoring Toolkit
https://dnsmonster.dev
GNU General Public License v3.0
316 stars 53 forks source link

Clickhouse output last batch #108

Open parsa97 opened 4 months ago

parsa97 commented 4 months ago

When using a pcap file, the last batch of data is missed because the context closes before it is sent. To solve this, I've created a new context for ClickHouse that remains open until the last batch is sent, and then it closes after the main context is closed.

Fix issue: #32

Tested with default configuration

Before Change

Dnsmonster log

metrics: {"clickhouseFailed":{"count":0},"clickhouseSentToOutput":{"count":3657536},"clickhouseSkipped":{"count":0},"packetLossPercent":{"value":0},"packetsCaptured":{"value":3668278},"packetsDropped":{"value":0},"packetsDuplicate":{"count":0},"packetsOverRatio":{"count":0}}

Clickhouse select count(*)

select count(*) from DNS_LOG
┌─count()─┐
│ 3650000 │
└─────────┘

After Change

Dnsmonster log

metrics: {"clickhouseFailed":{"count":0},"clickhouseSentToOutput":{"count":3657536},"clickhouseSkipped":{"count":0},"packetLossPercent":{"value":0},"packetsCaptured":{"value":3668278},"packetsDropped":{"value":0},"packetsDuplicate":{"count":0},"packetsOverRatio":{"count":0}}

Clikchouse select count(*)

select count(*) from DNS_LOG;
┌─count()─┐
│ 3657536 │
└─────────┘
netlify[bot] commented 4 months ago

Deploy Preview for unrivaled-llama-9deaa9 canceled.

Name Link
Latest commit af6e57add4ac8e27799b027fc1fe223aad547a93
Latest deploy log https://app.netlify.com/sites/unrivaled-llama-9deaa9/deploys/668e2fb10a617000084419f2