Support modules configuration

[dragoangel]

I think such exporter would be fit much better with /probe?module=rbl_ips&target=192.0.2.1 or /probe?module=rbl_domains&target=example.com, like it done in blackbox exporter, ssl exporter, etc. It will allow people who using stuff like Prometheus Operator control which things to monitor by simply proving CRDs in k8s to probe, instead of creating\adjusting configuration of exporter, this is much more dynamic flow. People without Prometheus Operator still can just edit prometheys.yaml without accessing and redeploying dnsbl exporter.

In this way dnsbl_exporter.conf configuration of module would have next scheme:

modules:
  <module_name_used_by_probe>: # meaning name of rbls to check with
    prober: dnsbl # name of modules that supported by exporter, if there no differnce between rbl resolving that should be treaded specially - this will be always dnsbl
    timeout: <duration> # module timeout
    dnsbl: # if not changed in prober, configures module behaviour
      # qeuery ips by default, f.e IPv4 192.0.2.1 would query 0.2.0.192.rbl-example.com
      # and IPv6 2001:db8:: - 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.rbl-example.com
      # if set to false query will be done as-is, f.e:
      # if target is mysite.com - then query is mysite.com.rbl-example.com.
      ips: true
      hashing: nil # defaults to no hashing, options: md5, sha1, sha256, etc.
      rbls:
        - <rbl.example.com>
        - <rbl2.example.com>
      resolvers: # if unset system resolvers will be used
        - <ns1.example.com>
        - <ns2.example.com>

resulting in:

modules:
  rbls_ips:
    prober: dnsbl
    timeout: 3s
    dnsbl:
      rbl_servers:
        - ix.dnsbl.manitu.net
        - zen.spamhaus.org
      resolvers:
        - 192.0.2.10
  rbl_domains:
    prober: dnsbl
    timeout: 3s
    dnsbl:
      ips: false
      rbl_servers:
        - dbl.spamhaus.org

Such workflow also as you see would allow using custom dns resolver per each module.

Here how would Prometheus probes would look like with probes flow:

apiVersion: monitoring.coreos.com/v1
kind: Probe
spec:
  interval: 15m
  jobName: myips
  module: rbl_ips
  prober:
    path: /probe
    scheme: http
    url: dnsbl-exporter.svc:9211
  scrapeTimeout: 5s
  targets:
    staticConfig:
      static:
        - 192.0.2.1
        - 192.0.2.2
        - 192.0.2.3
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
spec:
  interval: 15m
  jobName: mydomains
  module: rbl_domains
  prober:
    path: /probe
    scheme: http
    url: dnsbl-exporter.svc:9211
  scrapeTimeout: 5s
  targets:
    staticConfig:
      static:
        - example.com
        - example.org

[till]

Hey, there's already a probe endpoint, but I think it currently does not support domain based lookups.

[till]

I was wrong! :D When you start the exporter with --config.domain-based, then a request to /prober?target=x.y.z uses the domain based lookup. :)

Code is here: https://github.com/Luzilla/dnsbl_exporter/blob/main/internal/prober/prober.go

What changes would you suggest?

[dragoangel]

Hey, there's already a probe endpoint, but I think it currently does not support domain based lookups.

Hello @till, I checked usage at README.md and do not see anything which would mention such option or describe it's configuration flow.

[till]

Hey, there's already a probe endpoint, but I think it currently does not support domain based lookups.

Well I checked usage README.md and do not see anything which would mention such option or describe it's configuration flow.

No worries. I know not everything might be documented adequately. So assuming the exporter does what you need already, I would really like a PR adjusting the README or adding the required information to it. :)

[dragoangel]

Hey, there's already a probe endpoint, but I think it currently does not support domain based lookups.

Well I checked usage README.md and do not see anything which would mention such option or describe it's configuration flow.

No worries. I know not everything might be documented adequately. So assuming the exporter does what you need already, I would really like a PR adjusting the README or adding the required information to it. :)

Then usage modules part with configuration options would eliminate need of setting any extra argument flags and will not lock users to just one DNS server, as each module would have option to use own set of RBL and NS and in future if needed - more options. And will not require to have didicated deployment to query or domains or ips 😊, with one exporter you would have possibility to query any type of configuration.

[dragoangel]

@till already added ips to the top description, as this also is option to configure 😉 (better to name it ips, as rbls can contains ips, domains and other stuff like full emails even, and only ips are treated in reverse). Another non mandatory option would be hashing: nil, when rbl provide targets in way of md5 or sha1 hashed (domains, emails, etc), then user do not have to check what his domain would look like in md5 before creating target.

[till]

@dragoangel I am open to supporting your use cases. Do you want to start a PR?

The question of "multiple" resolvers comes up again and again. Still not entirely sure how to to implement that. I defer it to a local unbound.

For the hashing bit - I am not familiar with that. Do you have an example?

[dragoangel]

Hashing is rare thing, used to encode in rbl exactly links, like phishtank or emails, don't think someone would activelly used it 😅 in exporter, it was more to provide ideology on configuration I propose - that it allows to properly expand functionality without hardcoding things.

[dragoangel]

About multiply resolvers and PR, unfortunately can't say my Go skills (none) and free time would allow to create PRs 😓, but maybe this as options could help:

• https://stackoverflow.com/questions/59889882/specifying-dns-server-for-lookup-in-go
• https://github.com/Focinfi/go-dns-resolver

[dragoangel]

I renamed issue, to not confuse anybody :) as prober has a place

[dragoangel]

About documentation - I will deploy it and after checking how it goes will create PR with my findings to Readme

[dragoangel]

I'm also can provide PR to helm to support modules things if they would be in place 😊

[dragoangel]

@till /prober is not default thing to use, by default it should be /probe, and another thing: usually when exporter working with probes, default /metrics returns general details about exporter, which in case of dnsbl could be dns resolving time, resolving failures, go routines usage, exporter information (version, build), etc.

[dragoangel]

Another thing that could be cool in dnsbl_exporter.conf:

rbl_configuration:
  <rbl_domain_name>:
    ignored:
      - response: 127.0.0.7
        description: "whitelisted"
      - response_range: 127.0.0.100-127.0.0.115
    blocked:
      - response: 127.0.0.1
      - response_range: 127.0.0.120-127.0.0.135
    rate_limited:
      - response: 127.0.0.8
      - response_range: 127.0.0.200-127.0.0.215

This would allow to exclude false positive of luzilla_rbls_ips_blacklisted and will allow really rely on this metric for alerting globally, as now I can't say it's possible due to some rbls could reply 127.0.0.1 for rate_limited, f.e.: multi.uribl.com, and some rbls - for example score.senderscore.com provides "dynamic score from 0 to 100, have ability to set ranges would allow people to set threshold for what expect to be bad or not. Now to achieve same approach need to write own alerts per rbl with promql, not bad thing to do as well, but less user friendly, as such rbl_responses configuration could be contributed to the default yaml conf and work for everyone out of the box.

rate_limited stuff also would allow take some actions to administrators 😊, like reduce interval of probes or use another resolver...

if rbl is configured with at least one known response, and response returned that is not in ignored, blocked or rate_limited it treated as unknown and have dedicated metric which can be alerted with severity=info to notify adminstrator that they missed to configure something or their rbl added new response code which they not aware about.

optionally we can have description for any of status that would be added as a label to rbl with such response.

[dragoangel]

Also from what I can see IPv6 aren't supported, here: 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.6.f.d.1.f.7.8.e.0.0.3.0.0.2.zen.spamhaus.org IN A 127.0.0.11, but /prober?target=2003:e8:7f1d:f600:: return luzilla_rbls_errors{rbl="zen.spamhaus.org"} 1, expanding IPv6 to 2003:00e8:7f1d:f600:0000:0000:0000:0000 also doesn't help.

[till]

Yeah, don't know if spamhaus supports it. We can work on that in another ticket if there's demand.

[dragoangel]

Yeah, don't know if spamhaus supports it. We can work on that in another ticket if there's demand.

It is, you can check https://multirbl.valli.org/lookup/2003%3Ae8%3A7f1d%3A%3A.html and see dns query.

I now don't have ipv6 to monitor, tested out of curiosity 😄

[dragoangel]

Hi @till sorry for delay, week was quite busy, as promised created a PR to documentation: https://github.com/Luzilla/dnsbl_exporter/pull/227

Can you also please remove question tag from this issue as this is enhancement FR?

P.s. for IPv6 I created dedicated FR 😊