search

dnstap / golang-dnstap

flexible, structured event replication format for DNS servers (command-line tool and Golang package)
Apache License 2.0
132 stars 49 forks source link

Buffer does not get flushed if remote write target is unreachable for a long time #46

Open deathowl opened 2 years ago

deathowl commented 2 years ago

if the remote is unreachable that socket writer is trying to connect to as per : https://github.com/dnstap/golang-dnstap/blob/master/SocketWriter.go#L202 it just continues, leaving the buffer full 

2022/08/17 21:12:14 127.0.0.1:6000: open failed: dial tcp 127.0.0.1:6000: connect: connection refused < = failed write
E0817 21:12:17.800245  476695 logger.go:259] Failed to enqueue dnstap message type:CLIENT_QUERY  socket_family:INET6  socket_protocol:UDP  query_address:"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01"  response_address:"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01"  query_port:50564  response_port:8053  query_time_sec:1660767137  query_time_nsec:799106776  query_message:"\xd2U\x81\x05\x00\x01\x00\x00\x00\x00\x00\x01\x06github\x03com\x00\x00\x1c\x00\x01\x00\x00)\x10\x00\x00\x00\x00\x00\x00\x0c\x00\n\x00\x08y.5\x0f\x93~\xdb\xef" for sending, buffer is full
::1::11660767137AAAAgi <= failed subsequent enque attempt
cmikk commented 2 years ago

This is largely by design, and a common pattern you will find throughout the dnstap software stack.

A primary design goal of dnstap was to minimize impact on instrumented DNS servers. As such, if the consumer of dnstap data is unable to process said data, we would prefer it to signal to the producer to stop sending data rather than unproductively consume data, wasting the producer's time. The blocking provided by a full buffer is a very simple method of providing this feedback, at the cost of sending stale data when the pipeline eventually unsticks.