Intermittent Missing metrics

[LERUfic]

I think this is related to issue https://github.com/hiveco/conntrack_exporter/issues/3. But since It's already closed. I will open a new Issue.

Sometimes I only get these metrics

# HELP exposer_transferred_bytes_total Transferred bytes to metrics services
# TYPE exposer_transferred_bytes_total counter
exposer_transferred_bytes_total 1160731
# HELP exposer_scrapes_total Number of times metrics were scraped
# TYPE exposer_scrapes_total counter
exposer_scrapes_total 519
# HELP exposer_request_latencies Latencies of serving scrape requests, in microseconds
# TYPE exposer_request_latencies summary
exposer_request_latencies_count 519
exposer_request_latencies_sum 776717
exposer_request_latencies{quantile="0.5"} 2713
exposer_request_latencies{quantile="0.9"} 3124
exposer_request_latencies{quantile="0.99"} 3492

Conntrack Exporter that I use is v0.3.1 I'm running it on my GCE VM

I don't know if it's related but in debug log I only see this warning.

[DEBUG] WARNING: Tried to update an existing connection in our table but a match was not found (rebuilding=false):
    event=update     ipv4     2 tcp      6 120 TIME_WAIT src=10.0.0.1 dst=10.0.0.2 sport=64841 dport=9318 src=10.0.0.2 dst=10.0.0.1 sport=9318 dport=64841 [ASSURED]

[dbendelman]

Hi @LERUfic please provide the info requested in my comment here. Your issue could be the result of running inside docker without --net=host or proper permissions.

[LERUfic]

im not running it on docker. just directly execute the binary

root        2183 47.2 24.9 610780 500000 ?       Rl   Mar07 8066:07 ./conntrack_exporter-amd64-linux

I think this happened if the conntrack was left running for a long time.

[dbendelman]

Cannot reproduce:

$ uname -srv
Linux 6.5.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 12 10:22:43 UTC 2
$ wget -q https://github.com/hiveco/conntrack_exporter/releases/download/v0.3.1/conntrack_exporter-amd64-linux
$ chmod +x conntrack_exporter-amd64-linux
$ sudo ./conntrack_exporter-amd64-linux 
conntrack_exporter v0.3.1
Serving metrics at http://localhost:9318/metrics ...
$ wget -qO - http://localhost:9318/metrics
# HELP exposer_transferred_bytes_total Transferred bytes to metrics services
# TYPE exposer_transferred_bytes_total counter
exposer_transferred_bytes_total 1624
# HELP exposer_scrapes_total Number of times metrics were scraped
# TYPE exposer_scrapes_total counter
exposer_scrapes_total 2
# HELP exposer_request_latencies Latencies of serving scrape requests, in microseconds
# TYPE exposer_request_latencies summary
exposer_request_latencies_count 2
exposer_request_latencies_sum 225
exposer_request_latencies{quantile="0.5"} 106
exposer_request_latencies{quantile="0.9"} 106
exposer_request_latencies{quantile="0.99"} 106
# HELP conntrack_closing_connections How many connections to the remote host are currently closing?
# TYPE conntrack_closing_connections gauge
conntrack_closing_connections{host="127.0.0.1:9318"} 2

The last line shows it's exporting the connection table fine. I suspect this is an issue in your environment, eg. with permissions, a broken networking stack, etc. Try running it via docker and see if you get the same issue. Also note, you need to let it run for a few seconds before testing if it's working.

Closing as it seems to be a user issue.