search

c-mueller / ads

CoreDNS plugin to block ads. Inspried by PiHole.
Apache License 2.0
76 stars 12 forks source link

Blocklists not updating when using reload #47

Open b-m-f opened 3 years ago

b-m-f commented 3 years ago

Hi there,

I have noticed a problem when using https://coredns.io/plugins/reload/.

It seems that HTTP/S lists are not being updated on a reload. I dynamically inject lists into the Corefile, and lists seem to only become active on a restart, not a reload.

Maybe [INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist is related to this?

Let me know how I can help out to debug/fix this :)

Config

. {
    reload
    prometheus 0.0.0.0:9153
    ads {
      blacklist https://blocklistproject.github.io/List/ads.txt
      blacklist https://blocklistproject.github.io/Lists/abuse.txt
      blacklist https://blocklistproject.github.io/Lists/crypto.txt
      blacklist https://blocklistproject.github.io/Lists/drugs.txt
      blacklist https://blocklistproject.github.io/Lists/fraud.txt
      blacklist https://blocklistproject.github.io/Lists/gambling.txt
      blacklist https://blocklistproject.github.io/Lists/malware.txt
      blacklist https://blocklistproject.github.io/Lists/phishing.txt
      blacklist https://blocklistproject.github.io/Lists/porn.txt
      blacklist https://blocklistproject.github.io/Lists/ransomware.txt
      blacklist https://blocklistproject.github.io/Lists/redirect.txt
      blacklist https://blocklistproject.github.io/Lists/scam.txt
      blacklist https://blocklistproject.github.io/Lists/tiktok.txt
      blacklist https://blocklistproject.github.io/Lists/tracking.txt
      list-store /etc/coredns/blocklists
      default-lists

      target 127.0.0.1
      target-ipv6 ::1
    }
    forward . tls://1.1.1.1 {
       except fritz.box home lan local
       tls_servername cloudflare-dns.com
       health_check 5s
    }
    cache 30
}

Logs


[INFO] plugin/ads: Initializing CoreDNS 'ads' plugin. Version 0.2.5
[INFO] plugin/ads: Initializing CoreDNS 'ads' list update routines...
.:53
[INFO] plugin/reload: Running configuration MD5 = e60acf6ca0a2a149cfee6c1356874087
CoreDNS-1.8.3
linux/amd64, go1.15.8, 4293992b-dirty
[INFO] plugin/ads: Loaded Whitelist (HTTP) Length: 0
[INFO] plugin/ads: Loaded Blacklist (HTTP) Length: 2990552
[INFO] plugin/ads: Updating lists from HTTP URLs...
[INFO] plugin/ads: Scheduled next update in 23h53m16.626635257s
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist

----->> Blocklist gets removed from Corefile here!

[INFO] Reloading
[INFO] plugin/ads: Initializing CoreDNS 'ads' plugin. Version 0.2.5
[INFO] plugin/reload: Running configuration MD5 = 988a0e01e30fe01dfa814abd9f642fed
[INFO] Reloading complete

[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
c-mueller commented 3 years ago

Hey there, first of all thanks for submitting this issue.

I am assuming that this happens since the routine that looks for updates still awaits the initial interval of 24h after that the routine may download the updated lists based on the new set of URLs. To check this we have to reduce the auto-update-interval config option to a lower number such as 10 mins. Then the routine should update using the new list of course the Corefile has to be reloaded beforehand. If thats the case the fix should be fairly straightforward. If not further investigation is needed.

Unfortunately I am quite busy at the moment, which makes it hard for me to resolve this Problem. I will try my best though to investigate it ASAP.

b-m-f commented 3 years ago

Thanks. I will take a look at this this week and report back!

b-m-f commented 3 years ago

Hi @c-mueller ,

I have set auto-update-interval 2m

and waited for a while

[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] Reloading
[INFO] plugin/ads: Initializing CoreDNS 'ads' plugin. Version 0.2.5
[INFO] plugin/reload: Running configuration MD5 = 44f876db6c92812ad1432e045806e088
[INFO] Reloading complete
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
^N[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist

Unfortunately is does not seem to trigger the update

c-mueller commented 3 years ago

Okay, this will then need further investigation. I will take a look ASAP.

At the current state i cannot provide an ETA because i currently dont have an idea whats the cause here.

Just as a small sidenote: When initialy implementing the plugin i was unaware of the 'reload' plugin. This has not changed until now, however i think this is an important point i will have to take into account.

c-mueller commented 3 years ago

Just as a notice: I did not forget this. I am currently just not findig some free time to investigate it further.

b-m-f commented 3 years ago

Let me know when I can provide any more info to help out.

b-m-f commented 3 years ago

@c-mueller saw that you added the label. What kind of help do you need on this one?

c-mueller commented 3 years ago

Hey there, help wanted was more in the sense that someone else may investigate this, since I don't get to it at the morment.