search

publicarray / spksrc

Cross compilation framework to create native packages for the Synology's NAS
https://synocommunity.com/
Other
66 stars 14 forks source link

Generate Blacklist button seems to no longer work #23

Closed s1mwest closed 3 years ago

s1mwest commented 4 years ago

Setup

Package Name: dnscrypt-proxy Package Version: 2.0.44.-5

NAS Model: DS215J & DS218+ NAS Architecture: ARM375 & x64 (Celeron J3355) DSM version: 6.2.3-25426 update 2 (both devices)

Expected behavior

1) Affects both devices: I should not be able to resolve addresses that I have added to domains-blacklist-local-additions.txt 2) Affects only the DS218+: I should be able to successfully "generate blocklist" when I click the button

Actual behavior

1) It appears that pattern based blocking is working eg I cannot resolve ads.google.com due to "ads.*" but I have added mtalk.google.com to the local additions and this still seems to resolve - it didn't use to. This is happening on both my NAS devices (and query.log files show PASS instead of REJECT) 2) Clicking the "generate blocklist" button shows "changes saved" too quickly and the blocklist .txt file is not modified. This is only happening on the DS218+. The DS215J is updating (but continues to resolve specific addresses that I have added and show in the blocklist.txt file after generating)

Steps to reproduce

1. Install package 2. Configure .toml file, blacklist.conf file (enable blackist & whitelist & query logging) & add entries to domains-blacklist-local-additions.txt 3. Press "Generate Blacklist" button 4 Restart package 5 test

Package log

Check Package Center or /usr/local/{package}/var/

[2020-10-16 22:50:20] [NOTICE] -    32ms de.dnsmaschine.net                                                             [2020-10-16 22:50:20] [NOTICE] -    46ms dnscrypt.eu-dk                                                                 [2020-10-16 22:50:20] [NOTICE] -    53ms soltysiak                                                                      [2020-10-16 22:50:20] [NOTICE] -    67ms d0wn-is-ns2                                                                    [2020-10-16 22:50:21] [NOTICE] -    91ms ventricle.us                                                                   [2020-10-16 22:50:21] [NOTICE] -   158ms d0wn-tz-ns1                                                                    [2020-10-16 22:50:21] [NOTICE] -   192ms freetsa.org                                                                    [2020-10-16 22:50:21] [NOTICE] -   316ms publicarray-au                                                                 [2020-10-16 22:50:21] [NOTICE] Server with the lowest initial latency: dnscrypt.eu-nl (rtt: 29ms)                       [2020-10-16 22:50:21] [NOTICE] dnscrypt-proxy is ready - live servers: 10

Other logs

E.g. /var/log/messages or /var/log/synopkg.log

2020/10/16 22:48:00     stop dnscrypt-proxy 2.0.44-5 End start-stop-status stop ret=[0]                                 2020/10/16 22:48:00     stop dnscrypt-proxy 2.0.44-5 Begin unload apparmor                                              2020/10/16 22:48:00     stop dnscrypt-proxy 2.0.44-5 End unload apparmor ret=[0]                                        2020/10/16 22:48:01     stop dnscrypt-proxy: stop version 2.0.44-5 successfully, result 0                               2020/10/16 22:48:05     start dnscrypt-proxy: begin to start version 2.0.44-5                                           2020/10/16 22:48:06     start dnscrypt-proxy 2.0.44-5 Begin pre-load apparmor                                           2020/10/16 22:48:06     start dnscrypt-proxy 2.0.44-5 End pre-load apparmor ret=[0]                                     2020/10/16 22:48:06     start dnscrypt-proxy 2.0.44-5 Begin start-stop-status start                                     2020/10/16 22:48:06     start dnscrypt-proxy 2.0.44-5 End start-stop-status start ret=[0]                               2020/10/16 22:48:06     start dnscrypt-proxy: start version 2.0.44-5 successfully, result 0

Other Notes

These issues weren't present up until the last 7 days or so. This amazing product has been rock solid for me until then, and I am not certain at this stage that it is to be blamed. But I can't fathom what else in my environment could be the cause (actually it's driving me bonkers :D). I do regularly add/test entries in domains-blacklist-local-additions.txt and whitelist.txt (mostly for my smart TV's) but did perform a clean package install with minimal additions and still got these issues/symptoms.

Happy to try anything you suggest as I would really appreciate the help. Thanks in anticipation

publicarray commented 4 years ago

Thanks for such a detailed report!

In 2.0.44 there were some changes in how dnscrypt-proxy drops permissions. Would be willing to try the dev build here which removes the syntax check to work around the problem when saving files? https://github.com/publicarray/spksrc/releases/tag/dnscrypt-proxy-2.0.44_1 another workaround is to comment out user_name = and restart the package.

Please let me know if either of these work for you.

s1mwest commented 4 years ago

Hey, thank you too!

If I'm not mistaken, the link you provided is the version I have been using when this issue was first noticed,. The page states 2.0.44_1, but the assets show as 2.0.44-5.

Thanks, I commented out the user_name on the x64 device, restarted package but no change in behaviour..

It's odd that II have different symptoms on different devices and that I can't seem to roll either of them back to a working system, which raises my suspicions that something external to the package is causing these issues. So it may be a good idea if I simplify my setup for now to focus on one device and each issue separately.

publicarray commented 4 years ago

Thanks can you attach an example dnscrypt-proxy.toml and blacklist.conf file? I'll try if I can reproduce the issue on my end.

s1mwest commented 4 years ago

Just seen your response after typing this below - will attach on next post (sorry, it's late and I'm already in trouble :D)

OK a few observations to note: so far, whilst dealing with the x64 device only:

I have therefore re-installed 2.0.44-5 to my ARM375 device so I have resolution working. Need sleep haha will continue and update... will check that log file for package startup too

Weird :) but thanks again

publicarray commented 4 years ago

Sleep is important :)

it keeps stopping

Oh on a fresh install. What do the logs say?

Looks like you have some weird failures.

My thought for your blacklist issue might have been when a list server was down or changed location. Not sure if that could course an issue like this though.

s1mwest commented 4 years ago

Took a quick look in log on last startup attempt - I see this:

[2020-10-17 15:27:00] [NOTICE] dnscrypt-proxy 2.0.44 [2020-10-17 15:27:00] [NOTICE] Network connectivity detected [2020-10-17 15:27:00] [FATAL] listen udp 0.0.0.0:53: bind: address already in use

Can't quite understand what is using port 53 - I don't have any other DNS server running on that device.

Yes, I had added a lot of lists to my original setup, which did make me wonder about behaviour if any are unavailable. This is one of the reasons I stripped it back to defaults.

Attached files you requested. Thanks

x64 Device TOML file.txt

x63 Device Blacklist.conf.txt

s1mwest commented 4 years ago

Was in a rush earlier hence a few typo's above and lack of info such as; I change the default port listen_addresses to 53 in the TOML file FYI, other apps on this device are: AudioStation, DownloadStation, FileStation, MediaServer, oAuth Service, PhotoStation, PHP7.0, RadiusServer, Universal Search & VideoStation

EDIT: I've repeated the package removal, restarted device and installed package again (dnscrypt-proxy_x64-6.1_2.0.44-5.spk) - service is starting now on port 53 (this is the only change made to default install) and is stable(!). However, Generate Blocklist button is completing too quickly and blacklist.txt is still only showing the default pattern blocking only

EDIT: I forgot to uncomment blacklist_file = 'blacklist.txt' in TOML file. After doing this and restarting package, Generate Blocklist button/blacklist.txt behaviour still the same

EDIT: OK so after performing the same remove/install process on the other NAS (ARM375) the Generate Blocklist button is now behaving in the same way as the x64 device. At least I now have some consistency :) My current workaround is to put my most wanted blocked addresses directly into the blacklist.txt file on both devices.

Does the Generate Blocklist button create any logging that I could perhaps examine?

EDIT: Have just discovered that the [FATAL] listen udp 0.0.0.0:53: bind: address already in use error is shown after I comment out user_name = and restart the package. Re-enabling that setting then requires a restart of the device for the package to start successfully

publicarray commented 4 years ago

Thanks for these updates.

With ssh you can run it manually:

$ sudo su
$ cd /var/packages/dnscrypt-proxy/target/var/
$ ls -l # you may need this later to match the users:group
$ /usr/bin/env python generate-domains-blacklist.py > blacklist.txt.tmp && mv -f blacklist.txt.tmp blacklist.txt
Loading data from [file:domains-blacklist-local-additions.txt]
Loading data from [https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt]
Loading data from [https://mirror1.malwaredomains.com/files/justdomains]
Loading data from [https://www.malwaredomainlist.com/hostslist/hosts.txt]
Loading data from [https://easylist-downloads.adblockplus.org/easylist_noelemhide.txt]
Loading data from [https://easylist-downloads.adblockplus.org/easylistchina.txt]
Loading data from [https://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml]
Loading data from [http://sysctl.org/cameleon/hosts]
Loading data from [https://ssl.bblck.me/blacklists/domain-list.txt]
[https://ssl.bblck.me/blacklists/domain-list.txt] could not be loaded: <urlopen error [Errno -2] Name or service not known>

$ chown sc-dnscrypt-proxy:dnscrypt-proxy blacklist.txt

Removing https://ssl.bblck.me/blacklists/domain-list.txt from the list allows the script to run sucessfully.

Have just discovered that the [FATAL] listen udp 0.0.0.0:53: bind: address already in use error is shown after I comment out user_name = and restart the package. Re-enabling that setting then requires a restart of the device for the package to start successfully

That is definatly a bug, looks like the stopping script is not stopping dnscrypt-proxy properly. I'll have to look at it later.

publicarray commented 4 years ago

I think I need to expose the log of the script to the user somehow.

publicarray commented 4 years ago

FYI rather than a full restart you can kill dnscrypt-proxy pkill dnscrypt-proxy

s1mwest commented 4 years ago

Removing https://ssl.bblck.me/blacklists/domain-list.txt from the list allows the script to run sucessfully.

Yes, you've nailed it! I commented out that line in domains-blacklist.conf and the Generate Blocklist button worked. I'm not too hot with ssh, mostly using telnet to my NAS, so I haven't performed those steps, so thanks for the guidance.

Does this mean it's just unfortunate that an inactive/unresponsive URL breaks the entire generate process? If so, then I've been extremely lucky for a couple of years :D

EDIT: I see more clearly now; you've shown me how to diagnose the process, which is useful if any other lists fail to load

publicarray commented 4 years ago

No worries, I'm glad it worked 👍

I defenitly have to make this easier to dignose or send an error message to the user. Not everyone can use ssh/telnet to find the offending list.

Again thanks for your detailed responses, It really helps me and others when people google for the same problem.

s1mwest commented 4 years ago

All the thanks goes to you sir for such an awesome product, very prompt and friendly support as well as the improvements you continuously provide. The very least I can do to show my appreciation is report my findings and assist where I can.

s1mwest commented 4 years ago

Shortly after this was running smoothly again I had a repeated issue and discovered that https://someonewhocares.org/hosts/hosts is now also unavailable, so I commented this line to remedy.

I know this is a big ask, but could the order of processes within the generating blocklists task be modified to avoid blocklist generation failure? I believe what's happening is that an unavailable list causes the generate process to fail/stop, and as a result it seems to replace what was a previously populated blacklist.txt with nothing.

Perhaps the task can ignore unavailable lists (but log them)?

publicarray commented 3 years ago

@s1mwest I know it's been a hot while. I've now exposed the process output (stderr) to the GUI, so it makes diagnosing the issue much easier. The python script that processes the block list is from upstream I don't intent to improve it, but this makes it easy to find out what is going on.