search

letsencrypt / unbound_exporter

A Prometheus exporter for Unbound.
Apache License 2.0
178 stars 80 forks source link

unbound_exporter failed to scrape socket EOF #57

Closed sirizake closed 1 year ago

sirizake commented 1 year ago

I installed unbound_exporter: 0.4.1_8 on FreeBSD 13.2-RELEASE-p2 GENERIC amd64 and started service normally (unbound_exporter is running as pid 65551.) . However I see below error in unbound_exporter log hence unable to graph metrics in grafana ( grafana ID: 11705 ) (unbound dns version 1.17.1 running)

caller=unbound_exporter.go:461 Failedtoscrapesocket:=EOF caller=unbound_exporter.go:461 Failedtoscrapesocket:=EOF caller=unbound_exporter.go:461 Failedtoscrapesocket:=EOF

locally on the box i am able to run (curl 127.0.0.1:9167/metrics) and from a browser (curl X.X.X.X:9167/metrics) but output of curl 127.0.0.1:9167/metrics | grep '^unbound_up' unbound_up 0

Any assistance to resolve this error will be greatly appreciated

pgporada commented 1 year ago

@sirizake See if following the troubleshooting steps here helps you get further.

sirizake commented 1 year ago

Thanks @pgporada I followed the steps and re-checked unbound.conf but still no luck. See below

root@DDD:~ # service unbound_exporter restart 
Stopping unbound_exporter.
Waiting for PIDS: 65551.
Starting unbound_exporter.
root@DDD:~ # unbound_exporter -unbound.ca "" -unbound.cert ""
level=info ts=2023-09-19T16:17:22.904Z caller=unbound_exporter.go:480 Startingunbound_exporter=(MISSING)
level=info ts=2023-09-19T16:17:22.905Z caller=unbound_exporter.go:498 Listeningonaddress:port>=:9167
level=error ts=2023-09-19T16:17:22.905Z caller=unbound_exporter.go:499

root@DDD:~ # curl -s localhost:9167/metrics | grep ^unbound
unbound_up 0
root@DDD:~ #

logfile output

level=error ts=2023-09-19T16:22:45.174Z caller=unbound_exporter.go:461 Failedtoscrapesocket:=EOF
level=error ts=2023-09-19T16:23:00.174Z caller=unbound_exporter.go:461 Failedtoscrapesocket:=EOF
level=error ts=2023-09-19T16:23:15.172Z caller=unbound_exporter.go:461 Failedtoscrapesocket:=EOF
level=error ts=2023-09-19T16:23:30.182Z caller=unbound_exporter.go:461 Failedtoscrapesocket:=EOF
level=error ts=2023-09-19T16:23:45.173Z caller=unbound_exporter.go:461 Failedtoscrapesocket:=EOF
pgporada commented 1 year ago

Can you see metric output directly from unbound itself with unbound-control stats_noreset?

sirizake commented 1 year ago

yes @pgporada

output below

thread0.num.queries=830179
thread0.num.queries_ip_ratelimited=0
thread0.num.cachehits=799936
thread0.num.cachemiss=30243
thread0.num.prefetch=87252
thread0.num.expired=66826
thread0.num.recursivereplies=30242
thread0.num.dnscrypt.crypted=0
thread0.num.dnscrypt.cert=0
thread0.num.dnscrypt.cleartext=0
thread0.num.dnscrypt.malformed=0
thread0.requestlist.avg=5.16367
thread0.requestlist.max=34
thread0.requestlist.overwritten=0
thread0.requestlist.exceeded=0
thread0.requestlist.current.all=5
thread0.requestlist.current.user=1
thread0.recursion.time.avg=1.175753
thread0.recursion.time.median=0.082178
thread0.tcpusage=0
thread1.num.queries=269431
thread1.num.queries_ip_ratelimited=0
thread1.num.cachehits=255760
thread1.num.cachemiss=13671
thread1.num.prefetch=27915
thread1.num.expired=21389
thread1.num.recursivereplies=13670
thread1.num.dnscrypt.crypted=0
thread1.num.dnscrypt.cert=0
thread1.num.dnscrypt.cleartext=0
thread1.num.dnscrypt.malformed=0
thread1.requestlist.avg=2.06517
thread1.requestlist.max=27
thread1.requestlist.overwritten=0
thread1.requestlist.exceeded=0
thread1.requestlist.current.all=0
thread1.requestlist.current.user=0
thread1.recursion.time.avg=0.983659
thread1.recursion.time.median=0.147667
thread1.tcpusage=0
thread2.num.queries=153675
thread2.num.queries_ip_ratelimited=0
thread2.num.cachehits=147415
thread2.num.cachemiss=6260
thread2.num.prefetch=15852
thread2.num.expired=12122
thread2.num.recursivereplies=6260
thread2.num.dnscrypt.crypted=0
thread2.num.dnscrypt.cert=0
thread2.num.dnscrypt.cleartext=0
thread2.num.dnscrypt.malformed=0
thread2.requestlist.avg=2.25959
thread2.requestlist.max=20
thread2.requestlist.overwritten=0
thread2.requestlist.exceeded=0
thread2.requestlist.current.all=6
thread2.requestlist.current.user=1
thread2.recursion.time.avg=2.065191
thread2.recursion.time.median=0.106711
thread2.tcpusage=0
thread3.num.queries=572069
thread3.num.queries_ip_ratelimited=0
thread3.num.cachehits=550821
thread3.num.cachemiss=21248
thread3.num.prefetch=59855
thread3.num.expired=45915
thread3.num.recursivereplies=21245
thread3.num.dnscrypt.crypted=0
thread3.num.dnscrypt.cert=0
thread3.num.dnscrypt.cleartext=0
thread3.num.dnscrypt.malformed=0
thread3.requestlist.avg=3.33162
thread3.requestlist.max=30
thread3.requestlist.overwritten=0
thread3.requestlist.exceeded=0
thread3.requestlist.current.all=15
thread3.requestlist.current.user=2
thread3.recursion.time.avg=1.017278
thread3.recursion.time.median=0.0819776
thread3.tcpusage=1
thread4.num.queries=2641422
thread4.num.queries_ip_ratelimited=0
thread4.num.cachehits=2549157
thread4.num.cachemiss=92265
thread4.num.prefetch=275561
thread4.num.expired=210605
thread4.num.recursivereplies=92255
thread4.num.dnscrypt.crypted=0
thread4.num.dnscrypt.cert=0
thread4.num.dnscrypt.cleartext=0
thread4.num.dnscrypt.malformed=0
thread4.requestlist.avg=17.029
thread4.requestlist.max=62
thread4.requestlist.overwritten=0
thread4.requestlist.exceeded=0
thread4.requestlist.current.all=16
thread4.requestlist.current.user=4
thread4.recursion.time.avg=1.260211
thread4.recursion.time.median=0.0704264
thread4.tcpusage=1
thread5.num.queries=1906276
thread5.num.queries_ip_ratelimited=0
thread5.num.cachehits=1838890
thread5.num.cachemiss=67386
thread5.num.prefetch=197870
thread5.num.expired=151005
thread5.num.recursivereplies=67382
thread5.num.dnscrypt.crypted=0
thread5.num.dnscrypt.cert=0
thread5.num.dnscrypt.cleartext=0
thread5.num.dnscrypt.malformed=0
thread5.requestlist.avg=10.7433
thread5.requestlist.max=45
thread5.requestlist.overwritten=0
thread5.requestlist.exceeded=0
thread5.requestlist.current.all=37
thread5.requestlist.current.user=5
thread5.recursion.time.avg=1.169232
thread5.recursion.time.median=0.0714993
thread5.tcpusage=3
thread6.num.queries=607170
thread6.num.queries_ip_ratelimited=0
thread6.num.cachehits=585457
thread6.num.cachemiss=21713
thread6.num.prefetch=63598
thread6.num.expired=48610
thread6.num.recursivereplies=21712
thread6.num.dnscrypt.crypted=0
thread6.num.dnscrypt.cert=0
thread6.num.dnscrypt.cleartext=0
thread6.num.dnscrypt.malformed=0
thread6.requestlist.avg=3.41928
thread6.requestlist.max=26
thread6.requestlist.overwritten=0
thread6.requestlist.exceeded=0
thread6.requestlist.current.all=12
thread6.requestlist.current.user=2
thread6.recursion.time.avg=1.203429
thread6.recursion.time.median=0.0753828
thread6.tcpusage=0
thread7.num.queries=209960
thread7.num.queries_ip_ratelimited=0
thread7.num.cachehits=201853
thread7.num.cachemiss=8107
thread7.num.prefetch=21652
thread7.num.expired=16551
thread7.num.recursivereplies=8108
thread7.num.dnscrypt.crypted=0
thread7.num.dnscrypt.cert=0
thread7.num.dnscrypt.cleartext=0
thread7.num.dnscrypt.malformed=0
thread7.requestlist.avg=1.83655
thread7.requestlist.max=24
thread7.requestlist.overwritten=0
thread7.requestlist.exceeded=0
thread7.requestlist.current.all=0
thread7.requestlist.current.user=0
thread7.recursion.time.avg=1.457844
thread7.recursion.time.median=0.0906976
thread7.tcpusage=0
thread8.num.queries=437450
thread8.num.queries_ip_ratelimited=0
thread8.num.cachehits=421610
thread8.num.cachemiss=15840
thread8.num.prefetch=45653
thread8.num.expired=34918
thread8.num.recursivereplies=15839
thread8.num.dnscrypt.crypted=0
thread8.num.dnscrypt.cert=0
thread8.num.dnscrypt.cleartext=0
thread8.num.dnscrypt.malformed=0
thread8.requestlist.avg=3.1063
thread8.requestlist.max=34
thread8.requestlist.overwritten=0
thread8.requestlist.exceeded=0
thread8.requestlist.current.all=20
thread8.requestlist.current.user=1
thread8.recursion.time.avg=1.342415
thread8.recursion.time.median=0.0797679
thread8.tcpusage=1
thread9.num.queries=850375
thread9.num.queries_ip_ratelimited=0
thread9.num.cachehits=819225
thread9.num.cachemiss=31150
thread9.num.prefetch=88533
thread9.num.expired=67692
thread9.num.recursivereplies=31148
thread9.num.dnscrypt.crypted=0
thread9.num.dnscrypt.cert=0
thread9.num.dnscrypt.cleartext=0
thread9.num.dnscrypt.malformed=0
thread9.requestlist.avg=5.7364
thread9.requestlist.max=35
thread9.requestlist.overwritten=0
thread9.requestlist.exceeded=0
thread9.requestlist.current.all=2
thread9.requestlist.current.user=1
thread9.recursion.time.avg=1.177978
thread9.recursion.time.median=0.0803377
thread9.tcpusage=1
thread10.num.queries=5398115
thread10.num.queries_ip_ratelimited=0
thread10.num.cachehits=5211994
thread10.num.cachemiss=186121
thread10.num.prefetch=562886
thread10.num.expired=429984
thread10.num.recursivereplies=186117
thread10.num.dnscrypt.crypted=0
thread10.num.dnscrypt.cert=0
thread10.num.dnscrypt.cleartext=0
thread10.num.dnscrypt.malformed=0
thread10.requestlist.avg=29.8646
thread10.requestlist.max=79
thread10.requestlist.overwritten=0
thread10.requestlist.exceeded=0
thread10.requestlist.current.all=25
thread10.requestlist.current.user=8
thread10.recursion.time.avg=1.283834
thread10.recursion.time.median=0.0669418
thread10.tcpusage=2
thread11.num.queries=138789
thread11.num.queries_ip_ratelimited=0
thread11.num.cachehits=132948
thread11.num.cachemiss=5841
thread11.num.prefetch=14171
thread11.num.expired=10871
thread11.num.recursivereplies=5840
thread11.num.dnscrypt.crypted=0
thread11.num.dnscrypt.cert=0
thread11.num.dnscrypt.cleartext=0
thread11.num.dnscrypt.malformed=0
thread11.requestlist.avg=1.13977
thread11.requestlist.max=18
thread11.requestlist.overwritten=0
thread11.requestlist.exceeded=0
thread11.requestlist.current.all=14
thread11.requestlist.current.user=0
thread11.recursion.time.avg=0.924441
thread11.recursion.time.median=0.103634
thread11.tcpusage=0
thread12.num.queries=129449
thread12.num.queries_ip_ratelimited=0
thread12.num.cachehits=124055
thread12.num.cachemiss=5394
thread12.num.prefetch=13448
thread12.num.expired=10200
thread12.num.recursivereplies=5394
thread12.num.dnscrypt.crypted=0
thread12.num.dnscrypt.cert=0
thread12.num.dnscrypt.cleartext=0
thread12.num.dnscrypt.malformed=0
thread12.requestlist.avg=0.732672
thread12.requestlist.max=15
thread12.requestlist.overwritten=0
thread12.requestlist.exceeded=0
thread12.requestlist.current.all=0
thread12.requestlist.current.user=0
thread12.recursion.time.avg=0.688048
thread12.recursion.time.median=0.108716
thread12.tcpusage=0
thread13.num.queries=2341888
thread13.num.queries_ip_ratelimited=0
thread13.num.cachehits=2259220
thread13.num.cachemiss=82668
thread13.num.prefetch=245309
thread13.num.expired=187716
thread13.num.recursivereplies=82664
thread13.num.dnscrypt.crypted=0
thread13.num.dnscrypt.cert=0
thread13.num.dnscrypt.cleartext=0
thread13.num.dnscrypt.malformed=0
thread13.requestlist.avg=15.2079
thread13.requestlist.max=57
thread13.requestlist.overwritten=0
thread13.requestlist.exceeded=0
thread13.requestlist.current.all=23
thread13.requestlist.current.user=4
thread13.recursion.time.avg=1.088067
thread13.recursion.time.median=0.0729243
thread13.tcpusage=0
thread14.num.queries=207438
thread14.num.queries_ip_ratelimited=0
thread14.num.cachehits=199051
thread14.num.cachemiss=8387
thread14.num.prefetch=21769
thread14.num.expired=16615
thread14.num.recursivereplies=8388
thread14.num.dnscrypt.crypted=0
thread14.num.dnscrypt.cert=0
thread14.num.dnscrypt.cleartext=0
thread14.num.dnscrypt.malformed=0
thread14.requestlist.avg=1.19223
thread14.requestlist.max=16
thread14.requestlist.overwritten=0
thread14.requestlist.exceeded=0
thread14.requestlist.current.all=0
thread14.requestlist.current.user=0
thread14.recursion.time.avg=0.918769
thread14.recursion.time.median=0.101648
thread14.tcpusage=0
thread15.num.queries=384947
thread15.num.queries_ip_ratelimited=0
thread15.num.cachehits=369665
thread15.num.cachemiss=15282
thread15.num.prefetch=39890
thread15.num.expired=30459
thread15.num.recursivereplies=15279
thread15.num.dnscrypt.crypted=0
thread15.num.dnscrypt.cert=0
thread15.num.dnscrypt.cleartext=0
thread15.num.dnscrypt.malformed=0
thread15.requestlist.avg=2.42132
thread15.requestlist.max=20
thread15.requestlist.overwritten=0
thread15.requestlist.exceeded=0
thread15.requestlist.current.all=1
thread15.requestlist.current.user=1
thread15.recursion.time.avg=1.325853
thread15.recursion.time.median=0.0965287
thread15.tcpusage=0
thread16.num.queries=412236
thread16.num.queries_ip_ratelimited=0
thread16.num.cachehits=396058
thread16.num.cachemiss=16178
thread16.num.prefetch=42730
thread16.num.expired=32867
thread16.num.recursivereplies=16177
thread16.num.dnscrypt.crypted=0
thread16.num.dnscrypt.cert=0
thread16.num.dnscrypt.cleartext=0
thread16.num.dnscrypt.malformed=0
thread16.requestlist.avg=2.35863
thread16.requestlist.max=27
thread16.requestlist.overwritten=0
thread16.requestlist.exceeded=0
thread16.requestlist.current.all=1
thread16.requestlist.current.user=0
thread16.recursion.time.avg=0.910612
thread16.recursion.time.median=0.0926778
thread16.tcpusage=0
thread17.num.queries=1327290
thread17.num.queries_ip_ratelimited=0
thread17.num.cachehits=1280538
thread17.num.cachemiss=46752
thread17.num.prefetch=138708
thread17.num.expired=106299
thread17.num.recursivereplies=46746
thread17.num.dnscrypt.crypted=0
thread17.num.dnscrypt.cert=0
thread17.num.dnscrypt.cleartext=0
thread17.num.dnscrypt.malformed=0
thread17.requestlist.avg=7.96277
thread17.requestlist.max=56
thread17.requestlist.overwritten=0
thread17.requestlist.exceeded=0
thread17.requestlist.current.all=15
thread17.requestlist.current.user=3
thread17.recursion.time.avg=1.079104
thread17.recursion.time.median=0.0728507
thread17.tcpusage=0
thread18.num.queries=226399
thread18.num.queries_ip_ratelimited=0
thread18.num.cachehits=217467
thread18.num.cachemiss=8932
thread18.num.prefetch=23654
thread18.num.expired=18080
thread18.num.recursivereplies=8931
thread18.num.dnscrypt.crypted=0
thread18.num.dnscrypt.cert=0
thread18.num.dnscrypt.cleartext=0
thread18.num.dnscrypt.malformed=0
thread18.requestlist.avg=1.29298
thread18.requestlist.max=21
thread18.requestlist.overwritten=0
thread18.requestlist.exceeded=0
thread18.requestlist.current.all=6
thread18.requestlist.current.user=1
thread18.recursion.time.avg=0.796839
thread18.recursion.time.median=0.100381
thread18.tcpusage=0
thread19.num.queries=1399306
thread19.num.queries_ip_ratelimited=0
thread19.num.cachehits=1350000
thread19.num.cachemiss=49306
thread19.num.prefetch=145156
thread19.num.expired=111090
thread19.num.recursivereplies=49304
thread19.num.dnscrypt.crypted=0
thread19.num.dnscrypt.cert=0
thread19.num.dnscrypt.cleartext=0
thread19.num.dnscrypt.malformed=0
thread19.requestlist.avg=8.87385
thread19.requestlist.max=47
thread19.requestlist.overwritten=0
thread19.requestlist.exceeded=0
thread19.requestlist.current.all=19
thread19.requestlist.current.user=1
thread19.recursion.time.avg=1.091402
thread19.recursion.time.median=0.0702634
thread19.tcpusage=0
thread20.num.queries=163437
thread20.num.queries_ip_ratelimited=0
thread20.num.cachehits=156565
thread20.num.cachemiss=6872
thread20.num.prefetch=17101
thread20.num.expired=13195
thread20.num.recursivereplies=6872
thread20.num.dnscrypt.crypted=0
thread20.num.dnscrypt.cert=0
thread20.num.dnscrypt.cleartext=0
thread20.num.dnscrypt.malformed=0
thread20.requestlist.avg=0.842114
thread20.requestlist.max=15
thread20.requestlist.overwritten=0
thread20.requestlist.exceeded=0
thread20.requestlist.current.all=1
thread20.requestlist.current.user=0
thread20.recursion.time.avg=1.023456
thread20.recursion.time.median=0.110925
thread20.tcpusage=0
thread21.num.queries=138922
thread21.num.queries_ip_ratelimited=0
thread21.num.cachehits=133132
thread21.num.cachemiss=5790
thread21.num.prefetch=14275
thread21.num.expired=10979
thread21.num.recursivereplies=5790
thread21.num.dnscrypt.crypted=0
thread21.num.dnscrypt.cert=0
thread21.num.dnscrypt.cleartext=0
thread21.num.dnscrypt.malformed=0
thread21.requestlist.avg=0.774533
thread21.requestlist.max=17
thread21.requestlist.overwritten=0
thread21.requestlist.exceeded=0
thread21.requestlist.current.all=1
thread21.requestlist.current.user=0
thread21.recursion.time.avg=0.885239
thread21.recursion.time.median=0.105915
thread21.tcpusage=0
thread22.num.queries=163615
thread22.num.queries_ip_ratelimited=0
thread22.num.cachehits=156964
thread22.num.cachemiss=6651
thread22.num.prefetch=17172
thread22.num.expired=13084
thread22.num.recursivereplies=6650
thread22.num.dnscrypt.crypted=0
thread22.num.dnscrypt.cert=0
thread22.num.dnscrypt.cleartext=0
thread22.num.dnscrypt.malformed=0
thread22.requestlist.avg=1.27977
thread22.requestlist.max=19
thread22.requestlist.overwritten=0
thread22.requestlist.exceeded=0
thread22.requestlist.current.all=2
thread22.requestlist.current.user=1
thread22.recursion.time.avg=1.051817
thread22.recursion.time.median=0.101972
thread22.tcpusage=0
thread23.num.queries=400107
thread23.num.queries_ip_ratelimited=0
thread23.num.cachehits=385471
thread23.num.cachemiss=14636
thread23.num.prefetch=41433
thread23.num.expired=31809
thread23.num.recursivereplies=14636
thread23.num.dnscrypt.crypted=0
thread23.num.dnscrypt.cert=0
thread23.num.dnscrypt.cleartext=0
thread23.num.dnscrypt.malformed=0
thread23.requestlist.avg=3.08129
thread23.requestlist.max=27
thread23.requestlist.overwritten=0
thread23.requestlist.exceeded=0
thread23.requestlist.current.all=2
thread23.requestlist.current.user=0
thread23.recursion.time.avg=1.289723
thread23.recursion.time.median=0.0791542
thread23.tcpusage=0
thread24.num.queries=609056
thread24.num.queries_ip_ratelimited=0
thread24.num.cachehits=586909
thread24.num.cachemiss=22147
thread24.num.prefetch=63638
thread24.num.expired=48580
thread24.num.recursivereplies=22146
thread24.num.dnscrypt.crypted=0
thread24.num.dnscrypt.cert=0
thread24.num.dnscrypt.cleartext=0
thread24.num.dnscrypt.malformed=0
thread24.requestlist.avg=4.92197
thread24.requestlist.max=30
thread24.requestlist.overwritten=0
thread24.requestlist.exceeded=0
thread24.requestlist.current.all=4
thread24.requestlist.current.user=1
thread24.recursion.time.avg=1.234126
thread24.recursion.time.median=0.0778757
thread24.tcpusage=0
thread25.num.queries=509992
thread25.num.queries_ip_ratelimited=0
thread25.num.cachehits=490072
thread25.num.cachemiss=19920
thread25.num.prefetch=53076
thread25.num.expired=40331
thread25.num.recursivereplies=19919
thread25.num.dnscrypt.crypted=0
thread25.num.dnscrypt.cert=0
thread25.num.dnscrypt.cleartext=0
thread25.num.dnscrypt.malformed=0
thread25.requestlist.avg=3.29206
thread25.requestlist.max=29
thread25.requestlist.overwritten=0
thread25.requestlist.exceeded=0
thread25.requestlist.current.all=12
thread25.requestlist.current.user=0
thread25.recursion.time.avg=0.884476
thread25.recursion.time.median=0.0913615
thread25.tcpusage=0
thread26.num.queries=399476
thread26.num.queries_ip_ratelimited=0
thread26.num.cachehits=383595
thread26.num.cachemiss=15881
thread26.num.prefetch=41368
thread26.num.expired=31702
thread26.num.recursivereplies=15882
thread26.num.dnscrypt.crypted=0
thread26.num.dnscrypt.cert=0
thread26.num.dnscrypt.cleartext=0
thread26.num.dnscrypt.malformed=0
thread26.requestlist.avg=3.17478
thread26.requestlist.max=30
thread26.requestlist.overwritten=0
thread26.requestlist.exceeded=0
thread26.requestlist.current.all=1
thread26.requestlist.current.user=0
thread26.recursion.time.avg=0.977454
thread26.recursion.time.median=0.094931
thread26.tcpusage=0
thread27.num.queries=9263273
thread27.num.queries_ip_ratelimited=0
thread27.num.cachehits=8948375
thread27.num.cachemiss=314898
thread27.num.prefetch=962782
thread27.num.expired=734044
thread27.num.recursivereplies=314881
thread27.num.dnscrypt.crypted=0
thread27.num.dnscrypt.cert=0
thread27.num.dnscrypt.cleartext=0
thread27.num.dnscrypt.malformed=0
thread27.requestlist.avg=51.4949
thread27.requestlist.max=157
thread27.requestlist.overwritten=0
thread27.requestlist.exceeded=0
thread27.requestlist.current.all=68
thread27.requestlist.current.user=12
thread27.recursion.time.avg=1.301965
thread27.recursion.time.median=0.0241463
thread27.tcpusage=3
thread28.num.queries=693345
thread28.num.queries_ip_ratelimited=0
thread28.num.cachehits=668780
thread28.num.cachemiss=24565
thread28.num.prefetch=72200
thread28.num.expired=55142
thread28.num.recursivereplies=24564
thread28.num.dnscrypt.crypted=0
thread28.num.dnscrypt.cert=0
thread28.num.dnscrypt.cleartext=0
thread28.num.dnscrypt.malformed=0
thread28.requestlist.avg=3.9486
thread28.requestlist.max=32
thread28.requestlist.overwritten=0
thread28.requestlist.exceeded=0
thread28.requestlist.current.all=1
thread28.requestlist.current.user=0
thread28.recursion.time.avg=1.057385
thread28.recursion.time.median=0.0747997
thread28.tcpusage=0
thread29.num.queries=10950911
thread29.num.queries_ip_ratelimited=0
thread29.num.cachehits=10577862
thread29.num.cachemiss=373049
thread29.num.prefetch=1131809
thread29.num.expired=860908
thread29.num.recursivereplies=373048
thread29.num.dnscrypt.crypted=0
thread29.num.dnscrypt.cert=0
thread29.num.dnscrypt.cleartext=0
thread29.num.dnscrypt.malformed=0
thread29.requestlist.avg=56.9551
thread29.requestlist.max=137
thread29.requestlist.overwritten=0
thread29.requestlist.exceeded=0
thread29.requestlist.current.all=86
thread29.requestlist.current.user=13
thread29.recursion.time.avg=1.283643
thread29.recursion.time.median=0.0101346
thread29.tcpusage=2
thread30.num.queries=3656812
thread30.num.queries_ip_ratelimited=0
thread30.num.cachehits=3530238
thread30.num.cachemiss=126574
thread30.num.prefetch=382383
thread30.num.expired=292366
thread30.num.recursivereplies=126567
thread30.num.dnscrypt.crypted=0
thread30.num.dnscrypt.cert=0
thread30.num.dnscrypt.cleartext=0
thread30.num.dnscrypt.malformed=0
thread30.requestlist.avg=23.1415
thread30.requestlist.max=81
thread30.requestlist.overwritten=0
thread30.requestlist.exceeded=0
thread30.requestlist.current.all=29
thread30.requestlist.current.user=7
thread30.recursion.time.avg=1.286075
thread30.recursion.time.median=0.0684478
thread30.tcpusage=2
thread31.num.queries=898116
thread31.num.queries_ip_ratelimited=0
thread31.num.cachehits=864941
thread31.num.cachemiss=33175
thread31.num.prefetch=93782
thread31.num.expired=71839
thread31.num.recursivereplies=33179
thread31.num.dnscrypt.crypted=0
thread31.num.dnscrypt.cert=0
thread31.num.dnscrypt.cleartext=0
thread31.num.dnscrypt.malformed=0
thread31.requestlist.avg=5.46576
thread31.requestlist.max=31
thread31.requestlist.overwritten=0
thread31.requestlist.exceeded=0
thread31.requestlist.current.all=0
thread31.requestlist.current.user=0
thread31.recursion.time.avg=1.212711
thread31.recursion.time.median=0.081581
thread31.tcpusage=0
total.num.queries=48290926
total.num.queries_ip_ratelimited=0
total.num.cachehits=46594024
total.num.cachemiss=1696902
total.num.prefetch=5022481
total.num.expired=3833793
total.num.recursivereplies=1696835
total.num.dnscrypt.crypted=0
total.num.dnscrypt.cert=0
total.num.dnscrypt.cleartext=0
total.num.dnscrypt.malformed=0
total.requestlist.avg=30.9161
total.requestlist.max=157
total.requestlist.overwritten=0
total.requestlist.exceeded=0
total.requestlist.current.all=424
total.requestlist.current.user=69
total.recursion.time.avg=1.227502
total.recursion.time.median=0.0842017
total.tcpusage=16
time.now=1695143571.730576
time.up=92450.280987
time.elapsed=24348.917730
mem.cache.rrset=1140841081
mem.cache.message=570424929
mem.mod.iterator=16572
mem.mod.validator=1576792
mem.mod.respip=0
mem.cache.dnscrypt_shared_secret=0
mem.cache.dnscrypt_nonce=0
mem.mod.dynlibmod=0
mem.streamwait=0
mem.http.query_buffer=0
mem.http.response_buffer=0
histogram.000000.000000.to.000000.000001=757847
histogram.000000.000001.to.000000.000002=0
histogram.000000.000002.to.000000.000004=0
histogram.000000.000004.to.000000.000008=0
histogram.000000.000008.to.000000.000016=0
histogram.000000.000016.to.000000.000032=1
histogram.000000.000032.to.000000.000064=2
histogram.000000.000064.to.000000.000128=2
histogram.000000.000128.to.000000.000256=4
histogram.000000.000256.to.000000.000512=15
histogram.000000.000512.to.000000.001024=348
histogram.000000.001024.to.000000.002048=19401
histogram.000000.002048.to.000000.004096=22032
histogram.000000.004096.to.000000.008192=7890
histogram.000000.008192.to.000000.016384=10101
histogram.000000.016384.to.000000.032768=1252
histogram.000000.032768.to.000000.065536=3994
histogram.000000.065536.to.000000.131072=330215
histogram.000000.131072.to.000000.262144=321259
histogram.000000.262144.to.000000.524288=109889
histogram.000000.524288.to.000001.000000=36476
histogram.000001.000000.to.000002.000000=23124
histogram.000002.000000.to.000004.000000=30419
histogram.000004.000000.to.000008.000000=4523
histogram.000008.000000.to.000016.000000=4113
histogram.000016.000000.to.000032.000000=4173
histogram.000032.000000.to.000064.000000=2933
histogram.000064.000000.to.000128.000000=2425
histogram.000128.000000.to.000256.000000=2523
histogram.000256.000000.to.000512.000000=1578
histogram.000512.000000.to.001024.000000=252
histogram.001024.000000.to.002048.000000=33
histogram.002048.000000.to.004096.000000=11
histogram.004096.000000.to.008192.000000=0
histogram.008192.000000.to.016384.000000=0
histogram.016384.000000.to.032768.000000=0
histogram.032768.000000.to.065536.000000=0
histogram.065536.000000.to.131072.000000=0
histogram.131072.000000.to.262144.000000=0
histogram.262144.000000.to.524288.000000=0
num.query.type.TYPE0=235
num.query.type.A=33599677
num.query.type.NS=6419
num.query.type.CNAME=4533
num.query.type.SOA=25983
num.query.type.NULL=1
num.query.type.WKS=3894
num.query.type.PTR=275015
num.query.type.HINFO=3919
num.query.type.MX=329
num.query.type.TXT=291900
num.query.type.AAAA=1864585
num.query.type.SRV=257206
num.query.type.NAPTR=11916
num.query.type.DS=91
num.query.type.RRSIG=90
num.query.type.DNSKEY=87
num.query.type.SVCB=120638
num.query.type.HTTPS=11820432
num.query.type.ANY=2105
num.query.type.other=1400
num.query.class.CLASS0=9
num.query.class.IN=48288712
num.query.class.CH=89
num.query.class.CLASS6=8
num.query.class.CLASS16=1
num.query.class.ANY=2
num.query.class.other=1634
num.query.opcode.QUERY=48290455
num.query.tcp=1574655
num.query.tcpout=8316
num.query.udpout=18007584
num.query.tls=0
num.query.tls.resume=0
num.query.ipv6=0
num.query.https=0
num.query.flags.QR=0
num.query.flags.AA=0
num.query.flags.TC=0
num.query.flags.RD=48277798
num.query.flags.RA=267
num.query.flags.Z=1480
num.query.flags.AD=7158
num.query.flags.CD=1203
num.query.edns.present=574650
num.query.edns.DO=134098
num.answer.rcode.NOERROR=46930398
num.answer.rcode.FORMERR=407
num.answer.rcode.SERVFAIL=68256
num.answer.rcode.NXDOMAIN=1276104
num.answer.rcode.NOTIMPL=1537
num.answer.rcode.REFUSED=25536
num.answer.rcode.nodata=6715736
num.query.ratelimited=0
num.answer.secure=0
num.answer.bogus=0
num.rrset.bogus=0
num.query.aggressive.NOERROR=0
num.query.aggressive.NXDOMAIN=0
unwanted.queries=11379
unwanted.replies=63
msg.cache.count=2049912
rrset.cache.count=2442361
infra.cache.count=9897
key.cache.count=0
dnscrypt_shared_secret.cache.count=0
dnscrypt_nonce.cache.count=0
num.query.dnscrypt.shared_secret.cachemiss=0
num.query.dnscrypt.replay=0
num.query.authzone.up=0
num.query.authzone.down=0
pgporada commented 1 year ago

Great, can you post your unbound config please?

sirizake commented 1 year ago

please find below

server:
    verbosity: 1
    statistics-cumulative: no
    extended-statistics: yes
    num-threads: 32
    interface: X.X.X.X
        interface: A.B.C.D
    port: 53
    outgoing-interface: F.F.F.F
        outgoing-interface: K.K.K.K
    outgoing-range: 8192
    outgoing-num-tcp: 10000
    incoming-num-tcp: 10000
    msg-cache-size: 512m
    msg-cache-slabs: 32
    num-queries-per-thread: 4096
    rrset-cache-size: 1024m
    rrset-cache-slabs: 32
    cache-min-ttl: 300
    infra-cache-slabs: 32
    do-ip4: yes
    do-ip6: yes
    do-udp: yes
    do-tcp: yes
     access-control: A.B.C.D/24 allow
        access-control: B.B.B.B/22 allow
        access-control: D.D.D.D/19 allow
        chroot: ""
     username: "unbound"
    directory: "/usr/local/etc/unbound"
    logfile: "/var/log/unbound/unbound.log"
    use-syslog: no
    pidfile: "/usr/local/etc/unbound/unbound.pid"
    root-hints: "/usr/local/etc/unbound/root.hints"
    hide-identity: yes
    hide-version: yes
    harden-glue: yes
    harden-dnssec-stripped: yes
    qname-minimisation: yes
    use-caps-for-id: yes
    unwanted-reply-threshold: 10000000
    prefetch: yes
    prefetch-key: no
    deny-any: yes
    rrset-roundrobin: yes
    minimal-responses: yes
    serve-expired: yes
    val-log-level: 2
    key-cache-slabs: 16
    ratelimit-slabs: 32
    ip-ratelimit-slabs: 32
    control-enable: yes
    control-interface:X.X.X.X
    control-port: 8953
    control-use-cert: "no"
    server-key-file: "/usr/local/etc/unbound/unbound_server.key"
    server-cert-file: "/usr/local/etc/unbound/unbound_server.pem"
    control-key-file: "/usr/local/etc/unbound/unbound_control.key"
    control-cert-file: "/usr/local/etc/unbound/unbound_control.pem"
pgporada commented 1 year ago

By default unbound_exporter is looking for unbound-control at tcp://localhost:8953[1] and I notice that you redacted the control-interface in the unbound config. Can you show me whatever the freebsd version of netstat -plunt | grep 8953 is? Am I correct in assuming this is a pfsense box?

sirizake commented 1 year ago

This is a FreeBSD 13.2 box. when i run the command netstat -plunt | grep 8953 it returns netstat: lunt: unknown or uninstrumented protocol

however sockstat | grep 8953 returns:

unbound  unbound    66443 11 tcp4   127.0.0.1:8953        *:*
unbound  unbound    66443 12 tcp6   ::1:8953              *:*
unbound  unbound    66443 13 tcp4   myserverIP:8953     *:*
?        ?          ?     ?  tcp6   ::1:8953              ::1:30166
?        ?          ?     ?  tcp6   ::1:8953              ::1:18726
?        ?          ?     ?  tcp6   ::1:8953              ::1:34148
?        ?          ?     ?  tcp6   ::1:8953              ::1:12039

I temporarily enabled the other control interface to test 

 control-interface: 127.0.0.1
 control-interface: ::1
 control-interface: myserverIP

my error logs remain unchanged

level=error ts=2023-09-19T20:07:16.791Z caller=unbound_exporter.go:461 Failedtoscrapesocket:=EOF
level=error ts=2023-09-19T20:07:31.791Z caller=unbound_exporter.go:461 Failedtoscrapesocket:=EOF
level=error ts=2023-09-19T20:07:46.790Z caller=unbound_exporter.go:461 Failedtoscrapesocket:=EOF
level=error ts=2023-09-19T20:08:01.791Z caller=unbound_exporter.go:461 Failedtoscrapesocket:=EOF
level=error ts=2023-09-19T20:08:16.792Z caller=unbound_exporter.go:461 Failedtoscrapesocket:=EOF
level=error ts=2023-09-19T20:08:31.793Z caller=unbound_exporter.go:461 Failedtoscrapesocket:=EOF
level=error ts=2023-09-19T20:08:46.793Z caller=unbound_exporter.go:461 Failedtoscrapesocket:=EOF
jcjones commented 1 year ago

This looks like certs to me. Unbound's default certificate configs make Golang unhappy

sirizake commented 1 year ago

But control-use-cert: "no".

I will change this to yes and see what happens

sirizake commented 1 year ago

By default unbound_exporter is looking for unbound-control at tcp://localhost:8953[1] and I notice that you redacted the control-interface in the unbound config. Can you show me whatever the freebsd version of netstat -plunt | grep 8953 is? Am I correct in assuming this is a pfsense box?

@pgporada does this mean that i cannot query a remote host running unbound_exporter?

can i change the unbound-control at tcp://localhost:8953 to

unbound-control at tcp://A.B.C.D:8953

pgporada commented 1 year ago

Each unbound server we run also has the exporter alongside it. I'll have to set up a test because I don't know off the top of my head. My gut says it should work just fine. It'll take some time before I can get around to this - meaning not within the hour.

pgporada commented 1 year ago

Your unbound config is incorrect. You're missing the remote-control: top level header before all the control related options. See my following freebsd13 test unbound.conf for a working example. I used two freebsd13 VMs and was able to fetch metrics across the network as intended. Granted this is not using TLS because it's a quick canned test, but setting up the TLS certs is an easy task.

Network 1: 192.168.121.0/24 (default Vagrant network) Network 2: 192.168.33.0/24 (additional network) Host 1: 192.168.33.11/32 runs unbound Host 2: 192.168.33.12/32 runs unbound_exporter

unbound.conf

server:
        verbosity: 1
        statistics-cumulative: no
        extended-statistics: yes
        num-threads: 32
        port: 53
        outgoing-range: 8192
        outgoing-num-tcp: 10000
        incoming-num-tcp: 10000
        msg-cache-size: 512m
        msg-cache-slabs: 32
        num-queries-per-thread: 4096
        rrset-cache-size: 1024m
        rrset-cache-slabs: 32
        cache-min-ttl: 300
        infra-cache-slabs: 32
        do-ip4: yes
        do-ip6: yes
        do-udp: yes
        do-tcp: yes
        access-control: 127.0.0.0/8 allow
        access-control: 192.168.33.0/24 allow
        access-control: 192.168.121.0/24 allow
        chroot: ""
        username: "unbound"
        directory: "/usr/local/etc/unbound"
        logfile: "/var/log/unbound/unbound.log"
        use-syslog: no
        pidfile: "/usr/local/etc/unbound/unbound.pid"
        root-hints: "/usr/local/etc/unbound/root.hints"
        hide-identity: yes
        hide-version: yes
        harden-glue: yes
        harden-dnssec-stripped: yes
        qname-minimisation: yes
        use-caps-for-id: yes
        unwanted-reply-threshold: 10000000
        prefetch: yes
        prefetch-key: no
        deny-any: yes
        rrset-roundrobin: yes
        minimal-responses: yes
        serve-expired: yes
        val-log-level: 2
        key-cache-slabs: 16
        ratelimit-slabs: 32
        ip-ratelimit-slabs: 32

remote-control:
        control-enable: yes
        control-interface: 192.168.33.11
        control-interface: 192.168.121.24
        control-port: 8953
        control-use-cert: no
        server-key-file: "/usr/local/etc/unbound/unbound_server.key"
        server-cert-file: "/usr/local/etc/unbound/unbound_server.pem"
        control-key-file: "/usr/local/etc/unbound/unbound_control.key"
        control-cert-file: "/usr/local/etc/unbound/unbound_control.pem"

You can see unbound is listening on both configured networks:

$ sockstat | grep 8953
unbound  unbound    1082  7  tcp4   192.168.33.11:8953    *:*
unbound  unbound    1082  8  tcp4   192.168.121.24:8953   *:*

Now I make some queries against the Host 1 unbound server and see that unbound-control has metrics.

$ unbound-control stats_noreset | grep total.num.queries=
total.num.queries=8

Over on Host 2, I start a tmux session so I can temporarily run several unbound_exporter's in the foreground attempting to connect to each listening unbound-control port, and use curl in another pane to view the metrics. You don't need to use tmux, but it is a nice tool.

$ ./unbound_exporter -unbound.ca="" -unbound.cert="" -unbound.host="tcp://192.168.33.11:8953"
level=info ts=2023-09-21T14:35:17.400Z caller=unbound_exporter.go:509 Startingunbound_exporter=(MISSING)
level=info ts=2023-09-21T14:35:17.400Z caller=unbound_exporter.go:527 Listeningonaddress:port>=:9167
$ ./unbound_exporter -unbound.ca="" -unbound.cert="" -unbound.host="tcp://192.168.121.24:8953" -web.listen-address=":9168"
level=info ts=2023-09-21T14:42:55.806Z caller=unbound_exporter.go:509 Startingunbound_exporter=(MISSING)
level=info ts=2023-09-21T14:42:55.806Z caller=unbound_exporter.go:527 Listeningonaddress:port>=:9168

Because I have two exporters on Host 2 in the same network namespace, I have to configure different ports for them, but they're still scraping a single unbound server running on Host 1.

$ curl -s localhost:9167/metrics | egrep '^unbound_(up|query_types_total)'
unbound_query_types_total{type="A"} 8
unbound_up 1

$ curl -s localhost:9168/metrics | egrep '^unbound_(up|query_types_total)'
unbound_query_types_total{type="A"} 8
unbound_up 1
sirizake commented 1 year ago

many thanks @pgporada & @jcjones for your inputs. Graphs are now visible in grafana!

@pgporada remote control is part of my config. sorry i omitted that whiles copying it out

Below is a summary of modifications done:

vi /usr/local/etc/rc.d/unbound_exporter and update this line as shown below: (line 47 & 48)

${unbound_exporter_bind:+-web.listen-address $unbound_exporter_bin} \ ${unbound_exporter_path:+-web.telemetry-path $unbound_exporter_path} \

inside file /etc/rc.conf (added below lines, new unbound_exporter.log file was created in the location specified and chown unbound:unbound)

unbound_exporter_enable="YES" unbound_exporter_log_file="/var/log/unbound/unbound_exporter.log"

inside unbound.conf file // # vi /usr/local/etc/unbound/unbound.conf server extended-statistics: yes

updated the access control to include LAN for remote access access-control: X.X.X.X/24 allow

remote-control: control-enable: yes control-interface: Y.Y.Y.Y // include the loopback interface control-port: 8953 control-use-cert: "yes"

Also "unbound_control.key" , "unbound_control.pem" & "unbound_server.pem" were copied into /etc/unbound directory default location for unbound_exporter

Thanks again for your time and assistance. Greatly appreciated.

pgporada commented 1 year ago

Awesome, glad to hear it. Best of luck.

On Fri, Sep 22, 2023, 8:56 AM sirizake @.***> wrote:

many thanks @pgporada https://github.com/pgporada & @jcjones https://github.com/jcjones for your inputs. Graphs are now visible in grafana!

@pgporada https://github.com/pgporada remote control is part of my config. sorry i omitted that whiles copying it out

Below is a summary of modifications done: vi /usr/local/etc/rc.d/unbound_exporter and update this line as shown below: (line 47 & 48)

${unbound_exporter_bind:+-web.listen-address $unbound_exporter_bin} ${unbound_exporter_path:+-web.telemetry-path $unbound_exporter_path} \

inside file /etc/rc.conf (added below lines, new unbound_exporter.log file was created in the location specified and chown unbound:unbound)

unbound_exporter_enable="YES" unbound_exporter_log_file="/var/log/unbound/unbound_exporter.log"

inside unbound.conf file // # vi /usr/local/etc/unbound/unbound.conf server extended-statistics: yes

updated the access control to include LAN for remote access access-control: X.X.X.X/24 allow

remote-control: control-enable: yes control-interface: Y.Y.Y.Y // include the loopback interface control-port: 8953 control-use-cert: "yes"

Also "unbound_control.key" , "unbound_control.pem" & "unbound_server.pem" were copied into /etc/unbound directory default location for unbound_exporter

Thanks again for your time and assistance. Greatly appreciated.

— Reply to this email directly, view it on GitHub https://github.com/letsencrypt/unbound_exporter/issues/57#issuecomment-1731369589, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASFVZNSIAWRRCK52JLHHE3X3WDHVANCNFSM6AAAAAA46CPDEY . You are receiving this because you were mentioned.Message ID: @.***>