search

jacklul / pihole-updatelists

Update Pi-hole's lists from remote sources easily
MIT License
1.42k stars 82 forks source link

Container start not possible without changing the DNS servers temporarily #98

Closed D3rSchurk3 closed 2 years ago

D3rSchurk3 commented 2 years ago

The container needs a working dns server to start. If you use just one pihole and set it up as your dns server in router config it isn't reachable when the container is stopped. Therefore you need to change your router config temporarily every time you need to restart the container.

It's not such a big deal if you use two piholes, because one pihole is still reachable when you need to restart the other. But after a blackout or when you have to cut off the current, you would have to change the settings again.

Is there any way to avoid this?

jacklul commented 2 years ago

But isn't it also the case for the base pi-hole image? I specifically remember that the containers from Pi-hole team refused to start without working DNS.

Container start automatically updates the list and rebuilds gravity, this can't be done without working DNS.

D3rSchurk3 commented 2 years ago

No, using the base pi-hole image it works without a reachable dns server.

D3rSchurk3 commented 2 years ago

Wouldn't it possible to skip the update process after like 5-10 minutes or so when no dns server is reachable? Would be cool if there were a variable for that.

jacklul commented 2 years ago

Well then, back when I was doing the docker implementation that was how their container worked. I'll look into it when I have the time

D3rSchurk3 commented 2 years ago

That's cool, thank you very much! ;)

jacklul commented 2 years ago

Could you change your image to :dev and tell me if it works for you, did the script and gravity update is execute on boot? (/var/log/pihole-updatelists.log and /var/log/pihole_updateGravity.log inside container should have fresh data) I made a change in the start script but have no way to test it myself currently

D3rSchurk3 commented 2 years ago

Old version:

Added ENV to php: "PIHOLE_DOCKER_TAG" => "2022.01.1", "PHP_ERROR_LOG" => "/var/log/lighttpd/error.log", "ServerIP" => "192.168.1.131", "CORS_HOSTS" => "", "VIRTUAL_HOST" => "192.168.1.131", Using IPv4 and IPv6 ::: Preexisting ad list /etc/pihole/adlists.list detected ((exiting setup_blocklists early)) https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts ::: Testing pihole-FTL DNS: FTL started! ::: Testing lighttpd config: Syntax OK ::: All config checks passed, cleared for startup ... ::: Enabling Query Logging [i] Enabling logging...

[✓] Logging has been enabled! ::: Docker start setup complete Checking if custom gravity.db is set in /etc/pihole/pihole-FTL.conf Pi-hole version is v5.8.1 (Latest: v5.8.1) AdminLTE version is v5.10.1 (Latest: v5.10.1) FTL version is v5.13 (Latest: v5.13) Container tag is: 2022.01.1 [cont-init.d] 20-start.sh: exited 0. [cont-init.d] done. [services.d] starting services Starting lighttpd Starting crond Starting pihole-FTL (no-daemon) as pihole [services.d] done. Stopping pihole-FTL Stopping cron Stopping lighttpd [cont-finish.d] executing container finish scripts... [cont-finish.d] done. [s6-finish] waiting for services. [s6-finish] sending all processes the TERM signal. [s6-finish] sending all processes the KILL signal and exiting. [s6-init] making user provided files available at /var/run/s6/etc...exited 0. [s6-init] ensuring user provided files have correct perms...exited 0. [fix-attrs.d] applying ownership & permissions fixes... [fix-attrs.d] 01-resolver-resolv: applying... [fix-attrs.d] 01-resolver-resolv: exited 1. [fix-attrs.d] done. [cont-init.d] executing container initialization scripts... [cont-init.d] 10-pihole-updatelists.sh: executing... ownership of '/etc/pihole-updatelists/crontab' retained as root:root ownership of '/etc/pihole-updatelists/pihole-updatelists.conf' retained as root:root mode of '/etc/pihole-updatelists/crontab' retained as 0644 (rw-r--r--) mode of '/etc/pihole-updatelists/pihole-updatelists.conf' retained as 0644 (rw-r--r--)

  Pi-hole's Lists Updater by Jack'lul

https://github.com/jacklul/pihole-updatelists

Opened gravity database: /etc/pihole/gravity.db (23.04 MB)

Fetching ADLISTS from 'https://v.firebog.net/hosts/lists.php?type=tick'... Could not resolve host: v.firebog.net

Fetching WHITELIST from 'https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt'... Could not resolve host: raw.githubusercontent.com One of the lists failed to download, operation aborted!

Fetching REGEX_BLACKLIST from 'https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list'... Could not resolve host: raw.githubusercontent.com

Finished with 3 error(s) in 50.31 seconds. [cont-init.d] 10-pihole-updatelists.sh: exited 1. [cont-finish.d] executing container finish scripts... [cont-finish.d] done. [s6-finish] waiting for services. [s6-finish] sending all processes the TERM signal. [s6-finish] sending all processes the KILL signal and exiting. [s6-init] making user provided files available at /var/run/s6/etc...exited 0. [s6-init] ensuring user provided files have correct perms...exited 0. [fix-attrs.d] applying ownership & permissions fixes... [fix-attrs.d] 01-resolver-resolv: applying... [fix-attrs.d] 01-resolver-resolv: exited 1. [fix-attrs.d] done. [cont-init.d] executing container initialization scripts... [cont-init.d] 10-pihole-updatelists.sh: executing... ownership of '/etc/pihole-updatelists/crontab' retained as root:root ownership of '/etc/pihole-updatelists/pihole-updatelists.conf' retained as root:root mode of '/etc/pihole-updatelists/crontab' retained as 0644 (rw-r--r--) mode of '/etc/pihole-updatelists/pihole-updatelists.conf' retained as 0644 (rw-r--r--)

  Pi-hole's Lists Updater by Jack'lul

https://github.com/jacklul/pihole-updatelists

Opened gravity database: /etc/pihole/gravity.db (23.04 MB)

Fetching ADLISTS from 'https://v.firebog.net/hosts/lists.php?type=tick'... Could not resolve host: v.firebog.net

Fetching WHITELIST from 'https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt'... Could not resolve host: raw.githubusercontent.com One of the lists failed to download, operation aborted!

Fetching REGEX_BLACKLIST from 'https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list'... Could not resolve host: raw.githubusercontent.com

Finished with 3 error(s) in 45.29 seconds. [cont-init.d] 10-pihole-updatelists.sh: exited 1. [cont-finish.d] executing container finish scripts... [cont-finish.d] done. [s6-finish] waiting for services. [s6-finish] sending all processes the TERM signal. [s6-finish] sending all processes the KILL signal and exiting.

Dev version:

[s6-init] making user provided files available at /var/run/s6/etc...exited 0. [s6-init] ensuring user provided files have correct perms...exited 0. [fix-attrs.d] applying ownership & permissions fixes... [fix-attrs.d] 01-resolver-resolv: applying... [fix-attrs.d] 01-resolver-resolv: exited 1. [fix-attrs.d] done. [cont-init.d] executing container initialization scripts... [cont-init.d] 20-start.sh: executing... ::: Starting docker specific checks & setup for docker pihole/pihole

[i] Installing configs from /etc/.pihole... [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone! [i] Installing /etc/dnsmasq.d/01-pihole.conf... [✓] Installed /etc/dnsmasq.d/01-pihole.conf [i] Installing /etc/.pihole/advanced/06-rfc6761.conf... [✓] Installed /etc/dnsmasq.d/06-rfc6761.conf Existing DNS servers detected in setupVars.conf. Leaving them alone ::: Pre existing WEBPASSWORD found DNSMasq binding to default interface: eth0 Added ENV to php: "PIHOLE_DOCKER_TAG" => "2022.01.1", "PHP_ERROR_LOG" => "/var/log/lighttpd/error.log", "ServerIP" => "192.168.1.131", "CORS_HOSTS" => "", "VIRTUAL_HOST" => "192.168.1.131", Using IPv4 and IPv6 ::: Preexisting ad list /etc/pihole/adlists.list detected ((exiting setup_blocklists early)) https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts ::: Testing pihole-FTL DNS: FTL started! ::: Testing lighttpd config: Syntax OK ::: All config checks passed, cleared for startup ... ::: Enabling Query Logging [i] Enabling logging...

[✓] Logging has been enabled! ::: Docker start setup complete Checking if custom gravity.db is set in /etc/pihole/pihole-FTL.conf Pi-hole version is v5.8.1 (Latest: v5.8.1) AdminLTE version is v5.10.1 (Latest: v5.10.1) FTL version is v5.13 (Latest: v5.13) Container tag is: 2022.01.1 [cont-init.d] 20-start.sh: exited 0. [cont-init.d] 30-pihole-updatelists.sh: executing... removed '/etc/pihole-updatelists.conf' '/etc/pihole-updatelists.conf' -> '/etc/pihole-updatelists/pihole-updatelists.conf' removed '/etc/cron.d/pihole-updatelists' '/etc/cron.d/pihole-updatelists' -> '/etc/pihole-updatelists/crontab' ownership of '/etc/pihole-updatelists/crontab' retained as root:root ownership of '/etc/pihole-updatelists/pihole-updatelists.conf' retained as root:root mode of '/etc/pihole-updatelists/crontab' retained as 0644 (rw-r--r--) mode of '/etc/pihole-updatelists/pihole-updatelists.conf' retained as 0644 (rw-r--r--) Modifying /etc/cron.d/gravity-on-boot to delay the gravity update for 60 seconds so that pihole-updatelists has a chance to run before it! [cont-init.d] 30-pihole-updatelists.sh: exited 0. [cont-init.d] done. [services.d] starting services Starting pihole-FTL (no-daemon) as pihole Starting lighttpd Starting crond [services.d] done.

===========================================

So yes, the container starts. Do you need more information? I can't manage open the log files from inside the container.

jacklul commented 2 years ago

docker exec -it pihole bash will get you into container command line, then just check the log files. Replace pihole with container ID.

Gravity update should start about 1 minute after pihole-updatelists runs. (I add sleep 60 to the Pi-hole's crontab so that my script has a time to run)

D3rSchurk3 commented 2 years ago

pihole-updatelists.log:

Pi-hole's Lists Updater by Jack'lul https://github.com/jacklul/pihole-updatelists

Opened gravity database: /etc/pihole/gravity.db (23.04 MB)

Fetching ADLISTS from 'https://v.firebog.net/hosts/lists.php?type=tick'... done (29 entries) Processing... 29 exists

Fetching WHITELIST from 'https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt'... done Fetching WHITELIST from 'https://github.com/mmotti/pihole-regex/blob/master/whitelist.list'... done Merging multiple lists... done (1469 entries) Processing... 219 exists, 1247 invalid List of invalid entries: ...

Fetching REGEX_BLACKLIST from 'https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list'... done (14 entries) Processing... 14 exists

Ignored 1247 invalid entries. Finished successfully in 16.02 seconds.

======================================

pihole_updateGravity.log is empty

jacklul commented 2 years ago

Could you also check contents of the /etc/cron.d/gravity-on-boot file? Could you check in the webinterface did the gravity update actually happenend? (Adlists page, look around adlists last update time if it's around minute late than of pihole-updatelists.log)

By the way: Whitelist URL https://github.com/mmotti/pihole-regex/blob/master/whitelist.list is invalid, https://raw.githubusercontent.com/mmotti/pihole-regex/master/whitelist.list is the correct URL.

D3rSchurk3 commented 2 years ago

gravity-on-boot:

@reboot root PATH="$PATH:/usr/sbin:/usr/local/bin/" sleep 60 ; pihole updateGravity >/var/log/pihole_updateGravity.log || cat /var/log/pihole_updateGravity.log

No, the update did not happen. I've just rebuild the container using the dev image. Last update according to adlist page took place at about 5 hours ago whereas pihole-updateslists.log has just been updated when I've rebuild the container.

Thanks, just fixed the url ;)

D3rSchurk3 commented 2 years ago

For example for "https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts.txt"

dev image: Health status of this list: List unchanged upstream, Pi-hole used a local copy (OK) This list was added to Pi-hole 5 days ago (2022-01-08 14:44:13) Database entry was last modified 5 days ago (2022-01-08 14:44:13) The list contents were last updated 4 hours ago (2022-01-13 13:41:40) Number of domains on this list: 41792 Number of invalid domains on this list: 0 Database ID of this list: 34

latest image on another pi: Health status of this list: List unchanged upstream, Pi-hole used a local copy (OK) This list was added to Pi-hole a day ago (2022-01-12 18:35:46) Database entry was last modified a day ago (2022-01-12 18:35:46) The list contents were last updated an hour ago (2022-01-13 16:37:27) Number of domains on this list: 41846 Number of invalid domains on this list: 0 Database ID of this list: 2

jacklul commented 2 years ago

I messed up the placement of that sleep 60. Pushing update to dev image, could you update it, restart the container and verify the logs again?

D3rSchurk3 commented 2 years ago

Now it seems to be working:

gravity-on-boot: @reboot root sleep 60 ; PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updateGravity >/var/log/pihole_updateGravity.log || cat /var/log/pihole_updateGravity.log

pihole-updatelists.log: Pi-hole's Lists Updater by Jack'lul https://github.com/jacklul/pihole-updatelists

Opened gravity database: /etc/pihole/gravity.db (23.04 MB)

Fetching ADLISTS from 'https://v.firebog.net/hosts/lists.php?type=tick'... done (29 entries) Processing... 29 exists

Fetching WHITELIST from 'https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt'... done Fetching WHITELIST from 'https://raw.githubusercontent.com/mmotti/pihole-regex/master/whitelist.list'... done Merging multiple lists... done (197 entries) Processing... 197 exists

Fetching REGEX_BLACKLIST from 'https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list'... done (14 entries) Processing... 14 exists

Finished successfully in 11.06 seconds.

pihole-updateGravity.log: [i] Neutrino emissions detected... ^M^[[K [ ^|^s] Pulling blocklist source list into range

[i] Preparing new gravity database...^M^[[K [ ^|^s] Preparing new gravity database [i] Using libz compression

[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts [i] Status: Pending...^M^[[K [ ^|^s] Status: Retrieval successful [i] Analyzed 96293 domains [i] List stayed unchanged

[i] Target: https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts.txt [i] Status: Pending...^M^[[K [ ^|^s] Status: Retrieval successful [i] Analyzed 41846 domains [i] List has been updated ...

Health status of this list: List download was successful (OK) This list was added to Pi-hole 5 days ago (2022-01-08 14:44:13) Database entry was last modified 5 days ago (2022-01-08 14:44:13) The list contents were last updated 8 minutes ago (2022-01-13 17:48:15) Number of domains on this list: 41846 Number of invalid domains on this list: 0 Database ID of this list: 34

jacklul commented 2 years ago

Nice, I will push this to master soon.