search

jacklul / pihole-updatelists

Update Pi-hole's lists from remote sources easily
MIT License
1.39k stars 81 forks source link

pihole-updatelists unable to find host following initail install on host running unbound #144

Closed paulrobinsontkd closed 7 months ago

paulrobinsontkd commented 7 months ago

I have both pihole and unbound installed on the same docker image using https://github.com/chriscrowe/docker-pihole-unbound/tree/main/one-container

I built a custom docker image adding pihole-updatelists

It seems however the pihole-updatelists is failing to resolve hosts on subsequent reboots following initial install. The initial install worked fine and downloaded the additional block lists however on subsequent reboots of the docker container it is failing to resolve the host.

I think the issue is the ordering of services starting and potentially unbound not being started before it attempts to perform the updates. See log file below:-

services-up: info: copying legacy longrun unbound (no readiness notification) s6-rc: info: service legacy-services successfully started Starting unbound Disabled default gravity update schedule in /etc/cron.d/pihole

  Pi-hole's Lists Updater by Jack'lul

https://github.com/jacklul/pihole-updatelists

Opened gravity database: /etc/pihole/gravity.db (65.09 MB)

Fetching ADLISTS from 'https://v.firebog.net/hosts/lists.php?type=tick'... Could not resolve host: v.firebog.net

Fetching WHITELIST from 'https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt'... Could not resolve host: raw.githubusercontent.com

Fetching REGEX_BLACKLIST from 'https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list'... Could not resolve host: raw.githubusercontent.com

Finished with 3 error(s) in 30.04 seconds. [~] #

jacklul commented 7 months ago

The installer puts the execution of the script right before _postFTL, so at that point pihole-FTL should be running and for it to correctly resolve DNS you need to start unbound way earlier.

Some people simply added a fallback DNS setting to the docker compose and that fixed the startup issue:

dns:
  - 127.0.0.1
  - 1.1.1.1

I have no idea how I can fix this for you but it worked with base Pi-hole image last time I checked, perhaps Unbound takes too long to start and Pi-hole fails to resolve anything? Did you tried running just the linked docker image (without my script) and does the gravity update on start (_postFTL) complete or does it error out too?

paulrobinsontkd commented 7 months ago

I created a new docker image from the official pihole image just to rule it out and added unbound via docker file. When i start without your update script unbound and pihole services all start correctly

Starting unbound [i] Creating new gravity database [i] Migrating content of /etc/pihole/adlists.list into new database [✗] DNS resolution is currently unavailable [✓] DNS resolution is now available

[i] Neutrino emissions detected... [✓] Pulling blocklist source list into range

[✓] Preparing new gravity database [✓] Creating new gravity databases [i] Using libz compression

[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts [✓] Status: Retrieval successful [✓] Parsed 160200 exact domains and 0 ABP-style domains (ignored 1 non-domain entries) Sample of non-domain entries:

It takes anywhere from 10 - 15 seconds before unbound is started from the looks of it.

I then recreated the custom docker image based off official pihole adding in the update scripts to my dockerfile RUN wget -O - https://raw.githubusercontent.com/jacklul/pihole-updatelists/master/install.sh | bash -s docker

On first run unbound starts and all pihole services start before executing update script

Starting unbound Disabled default gravity update schedule in /etc/cron.d/pihole Gravity database not found - running 'pihole -g' command... [i] Creating new gravity database [i] Migrating content of /etc/pihole/adlists.list into new database [✗] DNS resolution is currently unavailable [✓] DNS resolution is now available

[i] Neutrino emissions detected... [✓] Pulling blocklist source list into range

[✓] Preparing new gravity database [✓] Creating new gravity databases [i] Using libz compression

[i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts [✓] Status: Retrieval successful [✓] Parsed 160200 exact domains and 0 ABP-style domains (ignored 1 non-domain entries) Sample of non-domain entries:

Opened gravity database: /etc/pihole/gravity.db (9.38 MB)

Fetching ADLISTS from 'https://v.firebog.net/hosts/lists.php?type=tick'... done (31 entries) Processing... 31 inserted

Fetching WHITELIST from 'https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt'... done (191 entries) Processing... 191 inserted

Fetching REGEX_BLACKLIST from 'https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list'... done (14 entries) Processing... 14 inserted

I restarted docker and the start up sequence seems different and the script looks to be executing prior to unbound starting

Starting unbound

  Pi-hole's Lists Updater by Jack'lul

https://github.com/jacklul/pihole-updatelists

Opened gravity database: /etc/pihole/gravity.db (65.11 MB)

Fetching ADLISTS from 'https://v.firebog.net/hosts/lists.php?type=tick'... Could not resolve host: v.firebog.net

Fetching WHITELIST from 'https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt'... Could not resolve host: raw.githubusercontent.com

Fetching REGEX_BLACKLIST from 'https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list'... Could not resolve host: raw.githubusercontent.com

Finished with 3 error(s) in 30.03 seconds.

I tried adding the dns flags to the docker compose file but it made zero difference.

Unfortunately my knowledge of debian services and timers is next to nothing, was looking if i can move unbound to start before pihole but have no idea how to do it.

I was hoping to have a custom docker image which installed pihole/unbound and update scripts which i could run once a new release comes out and saves me lots of effort post config. Might have to rethink and seperate out pihole and unbound if i can't get it to work.

jacklul commented 7 months ago

[✗] DNS resolution is currently unavailable [✓] DNS resolution is now available

I can see they are now waiting for the DNS to come up, I might add something like this to my script and hopefully that will fix this issue.

jacklul commented 7 months ago

Try using development branch, it will now wait for DNS pi.hole domain to be resolvable

paulrobinsontkd commented 7 months ago

Have deployed the script in a custom docker container with the following RUN statement using develop branch RUN wget -O - https://raw.githubusercontent.com/jacklul/pihole-updatelists/develop/install.sh | bash -s docker

I don't however think it has deployed the amended code as it is still failing.

I also do not see the echo's that are present in the updated docker.sh when pihole restarts for the second time echo "Waiting for DNS resolution to be available..."

Opened gravity database: /etc/pihole/gravity.db (65.13 MB)

Fetching ADLISTS from 'https://v.firebog.net/hosts/lists.php?type=tick'... Could not resolve host: v.firebog.net

Fetching WHITELIST from 'https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt'... Could not resolve host: raw.githubusercontent.com

Fetching REGEX_BLACKLIST from 'https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list'... Could not resolve host: raw.githubusercontent.com

Finished with 3 error(s) in 30.03 seconds.

[admin@qnap ~]# docker exec -it pihole bash root@pihole:/# cat /var/log/pihole-updatelists-boot.log [2023-11-30 17:59:46 UTC] [INFO] Pi-hole's Lists Updater by Jack'lul started [2023-11-30 17:59:46 UTC] [INFO] Opened gravity database: /etc/pihole/gravity.db (65.13 MB) [2023-11-30 17:59:46 UTC] [INFO] Fetching ADLISTS from 'https://v.firebog.net/hosts/lists.php?type=tick'... [2023-11-30 17:59:56 UTC] [ERROR] Could not resolve host: v.firebog.net [2023-11-30 17:59:56 UTC] [INFO] Fetching WHITELIST from 'https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt'... [2023-11-30 18:00:06 UTC] [ERROR] Could not resolve host: raw.githubusercontent.com [2023-11-30 18:00:06 UTC] [INFO] Fetching REGEX_BLACKLIST from 'https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list'... [2023-11-30 18:00:16 UTC] [ERROR] Could not resolve host: raw.githubusercontent.com [2023-11-30 18:00:16 UTC] [INFO] Finished with 3 error(s) in 30.03 seconds. root@pihole:/#

jacklul commented 7 months ago

Have deployed the script in a custom docker container with the following RUN statement using develop branch RUN wget -O - https://raw.githubusercontent.com/jacklul/pihole-updatelists/develop/install.sh | bash -s docker

That will still install from master branch, you gotta clone the repo locally and build the image from local filesystem.

You might also try using bash -s develop instead

paulrobinsontkd commented 7 months ago

Created local copy of develop branch and uninstalled then installed

root@pihole:/etc/pihole/pihole-updatelists-develop# bash install.sh 2023-11-30 19:12:05 URL:https://raw.githubusercontent.com/jacklul/pihole-updatelists/develop/pihole-updatelists.php [98944/98944] -> "/tmp/pihole-updatelists.php" [1] Backing up previous version... '/usr/local/sbin/pihole-updatelists' -> '/var/tmp/pihole-updatelists.old' mode of '/var/tmp/pihole-updatelists.old' retained as 0644 (rw-r--r--) renamed '/tmp/pihole-updatelists.php' -> '/usr/local/sbin/pihole-updatelists' mode of '/usr/local/sbin/pihole-updatelists' changed from 0644 (rw-r--r--) to 0755 (rwxr-xr-x) 2023-11-30 19:12:06 URL:https://raw.githubusercontent.com/jacklul/pihole-updatelists/develop/pihole-updatelists.bash [1235/1235] -> "/etc/bash_completion.d/pihole-updatelists" [1] Created crontab (/etc/cron.d/pihole-updatelists) mkdir: cannot create directory '/etc/pihole-updatelists': File exists root@pihole:/etc/pihole/pihole-updatelists-develop#

Still don't see any changes to behaviour upon second reboot of container

Starting unbound

  Pi-hole's Lists Updater by Jack'lul

https://github.com/jacklul/pihole-updatelists

Opened gravity database: /etc/pihole/gravity.db (65.14 MB)

Fetching ADLISTS from 'https://v.firebog.net/hosts/lists.php?type=tick'... Could not resolve host: v.firebog.net

Fetching WHITELIST from 'https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt'... Could not resolve host: raw.githubusercontent.com

[admin@qnap compose]#

AGain i don't see the echo's you added so don't think that code is deployed on my system

jacklul commented 7 months ago

mkdir: cannot create directory '/etc/pihole-updatelists': File exists

Installation failed here. Updated script gets installed after this step

Why do you have /etc/pihole-updatelists as file?

PS. Try wrapping logs in code blocks, will make your messages way more readable.

paulrobinsontkd commented 7 months ago

I deployed a fresh container which has the develop branch scripts deployed from dockerfile and my local repo

`
COPY install.sh docker.sh pihole-updatelists.* /tmp/pihole-updatelists/

RUN apt-get update && \
    apt-get install -Vy wget php-cli php-sqlite3 php-intl php-curl && \
    apt-get clean && \
    rm -fr /var/cache/apt/* /var/lib/apt/lists/*.lz4 && \
    chmod +x /tmp/pihole-updatelists/install.sh && \
    bash /tmp/pihole-updatelists/install.sh docker && \
    rm -fr /tmp/pihole-updatelists 
`

However it seems the code still doesn't work after initial install the subsequent restart fails to perform the check

`Starting unbound
Disabled default gravity update schedule in /etc/cron.d/pihole

      Pi-hole's Lists Updater by Jack'lul
 https://github.com/jacklul/pihole-updatelists

Opened gravity database: /etc/pihole/gravity.db (65.14 MB)

Fetching ADLISTS from 'https://v.firebog.net/hosts/lists.php?type=tick'... Could not resolve host: v.firebog.net

Fetching WHITELIST from 'https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt'... Could not resolve host: raw.githubusercontent.com

Fetching REGEX_BLACKLIST from 'https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list'... Could not resolve host: raw.githubusercontent.com

Finished with 3 error(s) in 30.03 seconds.

root@pihole:/# cat /var/log/pihole-updatelists-boot.logoot.log
[2023-11-30 20:24:40 UTC] [INFO]        Pi-hole's Lists Updater by Jack'lul started
[2023-11-30 20:24:40 UTC] [INFO]        Opened gravity database: /etc/pihole/gravity.db (65.14 MB)
[2023-11-30 20:24:40 UTC] [INFO]        Fetching ADLISTS from 'https://v.firebog.net/hosts/lists.php?type=tick'...
[2023-11-30 20:24:50 UTC] [ERROR]       Could not resolve host: v.firebog.net
[2023-11-30 20:24:50 UTC] [INFO]        Fetching WHITELIST from 'https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt'...
[2023-11-30 20:25:00 UTC] [ERROR]       Could not resolve host: raw.githubusercontent.com
[2023-11-30 20:25:00 UTC] [INFO]        Fetching REGEX_BLACKLIST from 'https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list'...
[2023-11-30 20:25:10 UTC] [ERROR]       Could not resolve host: raw.githubusercontent.com
[2023-11-30 20:25:10 UTC] [INFO]        Finished with 3 error(s) in 30.02 seconds.
root@pihole:/#`

I checked on the container and the new code is deployed in the script being run on startup

`root@pihole:/# cat /usr/local/bin/_updatelists.sh
#!/bin/bash
# This is the startup file for Docker installation that runs before actual _postFTL service is started

if [ ! -d "/etc/s6-overlay/s6-rc.d/_postFTL" ]; then
        echo "Missing /etc/s6-overlay/s6-rc.d/_postFTL - not a Docker installation?"
        exit
fi

# Respect PH_VERBOSE environment variable
if [ "${PH_VERBOSE:-0}" -gt 0 ]; then
        set -x
        SCRIPT_ARGS="--verbose --debug"
fi

# Recreate the config file if it is missing
if [ ! -f "/etc/pihole-updatelists/pihole-updatelists.conf" ]; then
        cp /etc/pihole-updatelists.conf /etc/pihole-updatelists/pihole-updatelists.conf
        echo "Created /etc/pihole-updatelists/pihole-updatelists.conf"
fi

# Fix permissions (when config directory is mounted as a volume)
chown root:root /etc/pihole-updatelists/*
chmod 644 /etc/pihole-updatelists/*

# Disable default gravity update schedule
if [ "$(grep 'pihole updateGravity' < /etc/cron.d/pihole | cut -c1-1)" != "#" ]; then
        sed -e '/pihole updateGravity/ s/^#*/#/' -i /etc/cron.d/pihole
        echo "Disabled default gravity update schedule in /etc/cron.d/pihole"
fi

# Create new schedule with random time
echo "#30 3 * * 6   root   /usr/bin/php /usr/local/sbin/pihole-updatelists --config=/etc/pihole-updatelists/pihole-updatelists.conf" > /etc/cron.d/pihole-updatelists
sed "s/#30 /$((1 + RANDOM % 58)) /" -i /etc/cron.d/pihole-updatelists

if [ -n "$SKIPGRAVITYONBOOT" ]; then
        echo "Lists update skipped - SKIPGRAVITYONBOOT=true"
else
        if [ ! -f "/etc/pihole/gravity.db" ]; then
                echo "Gravity database not found - running 'pihole -g' command..."
                pihole -g
        else
                if [ -z "$PHUL_SKIPDNSCHECK" ]; then
                        [ -n "$PHUL_DNSCHECK_DOMAIN" ] && CHECK_DOMAIN="$PHUL_DNSCHECK_DOMAIN" || CHECK_DOMAIN="pihole"
                        [ -n "$PHUL_DNSCHECK_TIMELIMIT" ] && CHECK_TIMELIMIT="$PHUL_DNSCHECK_TIMELIMIT" || CHECK_TIMELIMIT=300

                        COUNTER=0
                        while [ -z "$CHECK_IP" ] && [ "$COUNTER" -lt "$CHECK_TIMELIMIT" ]; do
                                CHECK_IP=$(nslookup "$CHECK_DOMAIN" | awk '/^Address: / { print $2 }')

                                if [ -z "$CHECK_IP" ]; then
                                        [ "$COUNTER" = 0 ] && echo "Waiting for DNS resolution to be available..."

                                        sleep 1
                                fi

                                ((COUNTER++))
                        done

                        [ -z "$CHECK_IP" ] && echo "Timed out while waiting for DNS resolution to be available"
                fi
        fi

        if [ -z "$(printenv PHUL_LOG_FILE)" ]; then
                export PHUL_LOG_FILE="-/var/log/pihole-updatelists-boot.log"
        fi

        # shellcheck disable=SC2086
        /usr/bin/php /usr/local/sbin/pihole-updatelists --config=/etc/pihole-updatelists/pihole-updatelists.conf --env --no-gravity --no-reload ${SCRIPT_ARGS}
fi
root@pihole:/#

`
jacklul commented 7 months ago

It probably was able to resolve pi.hole instantly so it produced no log message, maybe you should try resolving external domain - try setting in your docker compose PHUL_DNSCHECK_DOMAIN=firebog.net

paulrobinsontkd commented 7 months ago

@jacklul sorted, yes you was correct.

It would of likely been resolving pi.hole from the hosts file, when i changed the check to PHUL_DNSCHECK_DOMAIN=firebog.net It now waiting long enough for the DNS resolution to work and the update script executes fine.

Starting unbound
Waiting for DNS resolution to be available...

      Pi-hole's Lists Updater by Jack'lul
 https://github.com/jacklul/pihole-updatelists

Opened gravity database: /etc/pihole/gravity.db (65.14 MB)

Fetching ADLISTS from 'https://v.firebog.net/hosts/lists.php?type=tick'... done (31 entries)
Processing... 31 exists

Fetching WHITELIST from 'https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt'... done (191 entries)
Processing... 191 exists

Fetching REGEX_BLACKLIST from 'https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list'... done (14 entries)
Processing... 14 exists

Finished successfully in 0.3 seconds.

When i run a debug of unbound it seems it is reloading all the cache that is taking the time for the startup of DNS. Once up and running i don't expect the container to be stopped frequently but its nice to know that if it is restarted the update script will run correctly.

Many thanks for all your help and support to get this to work.

jacklul commented 7 months ago

I've modified the commit to use pi-hole.net by default and pushed to the master branch.

paulrobinsontkd commented 7 months ago

Thanks, have deployed from master and can confirm it is working 👍