search

jacklul / pihole-updatelists

Update Pi-hole's lists from remote sources easily
MIT License
1.39k stars 81 forks source link

Custom Docker - No files created in mapped /etc/updatelists folder #115

Closed mwatz1234 closed 1 year ago

mwatz1234 commented 1 year ago

When creating custom docker, no no files get created in the mapped /etc/updatelists folder (where within the docker container there is no files within /etc/pihole-updatelists.

The docker log shows this `` s6-rc: info: service s6rc-oneshot-runner: starting s6-rc: info: service s6rc-oneshot-runner successfully started s6-rc: info: service fix-attrs: starting s6-rc: info: service fix-attrs successfully started s6-rc: info: service legacy-cont-init: starting s6-rc: info: service legacy-cont-init successfully started s6-rc: info: service cron: starting s6-rc: info: service cron successfully started s6-rc: info: service _uid-gid-changer: starting s6-rc: info: service _uid-gid-changer successfully started s6-rc: info: service _startup: starting FTLCONF_REPLY_ADDR4 is deprecated. Converting to FTLCONF_LOCAL_IPV4 [i] Starting docker specific checks & setup for docker pihole/pihole [i] Setting capabilities on pihole-FTL where possible [i] Applying the following caps to pihole-FTL:

When building the custom docker see the following ``

34 [linux/arm64 5/6] RUN wget -O - https://raw.githubusercontent.com/jacklul/pihole-updatelists/master/install.sh | bash /dev/stdin docker

34 0.462 --2022-08-29 00:09:57-- https://raw.githubusercontent.com/jacklul/pihole-updatelists/master/install.sh

34 0.735 Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...

34 0.791 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.

34 0.966 HTTP request sent, awaiting response... 200 OK

34 0.973 Length: 7590 (7.4K) [text/plain]

34 0.988 Saving to: 'STDOUT'

34 1.004

34 1.004 0K ....... 100% 223K=0.03s

34 1.029

34 1.031 2022-08-29 00:09:57 (223 KB/s) - written to stdout [7590/7590]

34 1.031

34 2.878 2022-08-29 00:09:59 URL:https://raw.githubusercontent.com/jacklul/pihole-updatelists/master/pihole-updatelists.php [95372/95372] -> "/usr/local/sbin/pihole-updatelists" [1]

34 2.986 mode of '/usr/local/sbin/pihole-updatelists' changed from 0644 (rw-r--r--) to 0755 (rwxr-xr-x)

34 3.611 2022-08-29 00:10:00 URL:https://raw.githubusercontent.com/jacklul/pihole-updatelists/master/pihole-updatelists.conf [669/669] -> "/etc/pihole-updatelists.conf" [1]

34 4.185 2022-08-29 00:10:01 URL:https://raw.githubusercontent.com/jacklul/pihole-updatelists/master/pihole-updatelists.bash [1152/1152] -> "/etc/bash_completion.d/pihole-updatelists" [1]

34 4.365 Created crontab (/etc/cron.d/pihole-updatelists)

34 4.474 mkdir: created directory '/etc/pihole-updatelists'

34 4.564 mkdir: created directory '/etc/s6-overlay/s6-rc.d/_updatelistsonboot'

34 4.579 Installed container service files!

34 4.588 Appended dependency to _gravityonboot service (/etc/s6-overlay/s6-rc.d/_gravityonboot/dependencies)!

34 DONE 4.6s

``

I do see pihole-updatelists.conf in the /etc folder within the docker container

jacklul commented 1 year ago

There should be /var/log/pihole-updatelists-onboot.log log file indicating the start script ran, if it didn't then initial setup (and moving all files to /etc/pihole-updatelists) did not happen. Does this happen with my image or only custom docker approach? I can see on the log that the service "_updatelistsonboot" did not start, something went wrong while building the custom image, are you sure you're pulling it correctly into your setup? Try using dev branch I just pushed to

mwatz1234 commented 1 year ago

Using my customer docker (pointing to your master), i don't see the log file /var/log/pihole-updatelists-onboot.log.

Using your docker image, i don't see the log file (/var/log/pihole-updatelists-onboot.log), but I do see the /etc/pihole-updatelists has the file pihole-updatelists.conf. However editing that file, and running pihole-updatelists on bash within the container says there is not remote lists set in the configuration file. However there is pihole-updatelists.conf file at /etc path, and if I edit that file, and run pihole-updatelists within the docker container bash, I do see it updating the lists as expected.

When I use my customer docker, pointing to your developer branch (RUN wget -O - https://raw.githubusercontent.com/jacklul/pihole-updatelists/develop/install.sh | bash /dev/stdin docker) I get the same issue as above. As in I see the .conf file, but editing that one (etc/pihole-updatelists/pihole-updatelists.conf) does nothing, and only editing etc/pihole-updatelists.conf will take. As well there is no log file /var/log/pihole-updatelists-onboot.log

If I create a docker with just pihole\pihole, and bash into that container and install your deloper branch, I don't get the conf file at etc/pihole-updatelists/pihole-updatelists.conf (the path exists, just not the file), as the conf file is only at location etc/pihole-updatelists.conf, as well I don't get the log file (/var/log/pihole-updatelists-onboot.log) ` root@pihole:/# apt-get update && apt-get install -y sudo bash nano curl wget php-cli php-sqlite3 php-intl php-curl Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB] Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB] Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB] Get:4 http://deb.debian.org/debian bullseye/main arm64 Packages [8069 kB] Get:5 http://deb.debian.org/debian-security bullseye-security/main arm64 Packages [178 kB] Get:6 http://deb.debian.org/debian bullseye-updates/main arm64 Packages [2604 B] Fetched 8458 kB in 4s (2380 kB/s) Reading package lists... Done Reading package lists... Done Building dependency tree... Done Reading state information... Done bash is already the newest version (5.1-2+deb11u1). bash set to manually installed. php-intl is already the newest version (2:7.4+76). php-sqlite3 is already the newest version (2:7.4+76). sudo is already the newest version (1.9.5p2-3). curl is already the newest version (7.74.0-1.3+deb11u2). The following additional packages will be installed: php7.4-curl Suggested packages: hunspell The following NEW packages will be installed: nano php-cli php-curl php7.4-curl wget 0 upgraded, 5 newly installed, 0 to remove and 1 not upgraded. Need to get 1634 kB of archives. After this operation, 6307 kB of additional disk space will be used. Get:1 http://deb.debian.org/debian bullseye/main arm64 nano arm64 5.4-2+deb11u1 [646 kB] Get:2 http://deb.debian.org/debian bullseye/main arm64 wget arm64 1.21-1+deb11u1 [946 kB] Get:3 http://deb.debian.org/debian bullseye/main arm64 php-cli all 2:7.4+76 [6796 B] Get:4 http://deb.debian.org/debian-security bullseye-security/main arm64 php7.4-curl arm64 7.4.30-1+deb11u1 [29.1 kB] Get:5 http://deb.debian.org/debian bullseye/main arm64 php-curl all 2:7.4+76 [6364 B] Fetched 1634 kB in 0s (5503 kB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package nano. (Reading database ... 11639 files and directories currently installed.) Preparing to unpack .../nano_5.4-2+deb11u1_arm64.deb ... Unpacking nano (5.4-2+deb11u1) ... Selecting previously unselected package wget. Preparing to unpack .../wget_1.21-1+deb11u1_arm64.deb ... Unpacking wget (1.21-1+deb11u1) ... Selecting previously unselected package php-cli. Preparing to unpack .../php-cli_2%3a7.4+76_all.deb ... Unpacking php-cli (2:7.4+76) ... Selecting previously unselected package php7.4-curl. Preparing to unpack .../php7.4-curl_7.4.30-1+deb11u1_arm64.deb ... Unpacking php7.4-curl (7.4.30-1+deb11u1) ... Selecting previously unselected package php-curl. Preparing to unpack .../php-curl_2%3a7.4+76_all.deb ... Unpacking php-curl (2:7.4+76) ... Setting up wget (1.21-1+deb11u1) ... Setting up php7.4-curl (7.4.30-1+deb11u1) ...

Creating config file /etc/php/7.4/mods-available/curl.ini with new version Setting up nano (5.4-2+deb11u1) ... update-alternatives: using /bin/nano to provide /usr/bin/editor (editor) in auto mode update-alternatives: warning: skip creation of /usr/share/man/man1/editor.1.gz because associated file /usr/share/man/man1/nano.1.gz (of link group editor) doesn't exist update-alternatives: using /bin/nano to provide /usr/bin/pico (pico) in auto mode update-alternatives: warning: skip creation of /usr/share/man/man1/pico.1.gz because associated file /usr/share/man/man1/nano.1.gz (of link group pico) doesn't exist Setting up php-cli (2:7.4+76) ... update-alternatives: using /usr/bin/php.default to provide /usr/bin/php (php) in auto mode update-alternatives: warning: skip creation of /usr/share/man/man1/php.1.gz because associated file /usr/share/man/man1/php.default.1.gz (of link group php) doesn't exist update-alternatives: using /usr/bin/phar.default to provide /usr/bin/phar (phar) in auto mode update-alternatives: warning: skip creation of /usr/share/man/man1/phar.1.gz because associated file /usr/share/man/man1/phar.default.1.gz (of link group phar) doesn't exist update-alternatives: using /usr/bin/phar.phar.default to provide /usr/bin/phar.phar (phar.phar) in auto mode update-alternatives: warning: skip creation of /usr/share/man/man1/phar.phar.1.gz because associated file /usr/share/man/man1/phar.phar.default.1.gz (of link group phar.phar) doesn't exist Setting up php-curl (2:7.4+76) ... Processing triggers for php7.4-cli (7.4.30-1+deb11u1) ... Processing triggers for php7.4-cgi (7.4.30-1+deb11u1) ... root@pihole:/# wget -O - https://raw.githubusercontent.com/jacklul/pihole-updatelists/develop/install.sh | bash /dev/stdin docker --2022-09-01 11:30:53-- https://raw.githubusercontent.com/jacklul/pihole-updatelists/develop/install.sh Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.110.133, 185.199.109.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 7329 (7.2K) [text/plain] Saving to: 'STDOUT'

2022-09-01 11:30:53 (1.12 MB/s) - written to stdout [7329/7329]

2022-09-01 11:30:53 URL:https://raw.githubusercontent.com/jacklul/pihole-updatelists/master/pihole-updatelists.php [95372/95372] -> "/usr/local/sbin/pihole-updatelists" [1] mode of '/usr/local/sbin/pihole-updatelists' changed from 0644 (rw-r--r--) to 0755 (rwxr-xr-x) 2022-09-01 11:30:54 URL:https://raw.githubusercontent.com/jacklul/pihole-updatelists/master/pihole-updatelists.conf [669/669] -> "/etc/pihole-updatelists.conf" [1] 2022-09-01 11:30:54 URL:https://raw.githubusercontent.com/jacklul/pihole-updatelists/master/pihole-updatelists.bash [1152/1152] -> "/etc/bash_completion.d/pihole-updatelists" [1] Created crontab (/etc/cron.d/pihole-updatelists) mkdir: created directory '/etc/pihole-updatelists' mkdir: created directory '/etc/s6-overlay/s6-rc.d/_updatelistsonboot' mode of '/usr/local/bin/_updatelistsonboot.sh' changed from 0644 (rw-r--r--) to 0755 (rwxr-xr-x) Installed container service files! Added dependency to _gravityonboot service (/etc/s6-overlay/s6-rc.d/_gravityonboot/dependencies.d/_updatelistsonboot)!`

jacklul commented 1 year ago

It should not use /etc/pihole-updatelists.conf at all -> https://github.com/jacklul/pihole-updatelists/blob/master/install.sh#L182

seanob86 commented 1 year ago

Appears I am having similar issue as well with custom docker image.

I just modified my crontab file on the host and restarted container but the changes aren't being synced within the container.

In older versions of pihole image (using custom docker image) I could see in the docker log during startup the crontab file being linked to /etc/cron.d/pihole-updatelists.

Now the container is just using the default /etc/cron.d/pihole-updatelists cron.

In /var/log/ there is no /var/log/pihole-update lists-on boot.log

EDIT: Looking at the recent changes in install.sh the following code has been removed ln -sv /etc/pihole-updatelists/crontab /etc/cron.d/pihole-updatelists. I'm not sure if that should be executed somewhere else in the script??

jacklul commented 1 year ago

In older versions of pihole image (using custom docker image) I could see in the docker log during startup the crontab file being linked to /etc/cron.d/pihole-updatelists.

Now the container is just using the default /etc/cron.d/pihole-updatelists cron.

In /var/log/ there is no /var/log/pihole-update lists-on boot.log

EDIT: Looking at the recent changes in install.sh the following code has been removed ln -sv /etc/pihole-updatelists/crontab /etc/cron.d/pihole-updatelists. I'm not sure if that should be executed somewhere else in the script??

I was instructed on DMs to never do this kind of stuff while creating docker images because of a risk of having dead links when something changes. Pi-hole's image is using the same approach - you can add extra step on the bottom of your custom Dockerfile that modifies those files.

Pushed small change to the dev branch.

seanob86 commented 1 year ago

Dev is now logging to pihole-updatelists-onboot.log.

mwatz1234 commented 1 year ago

Using the latest devloper branch, I get the log file now, but it seems that there needs to be a delay now for pihole-updatelists to start, or wait till gravity database is created.

I get the "Lists update skipped - gravity database not found" message, which means updatelists is not using /etc/pihole-updatelists/pihole-updatelists.conf file, still using /etc/pihole-updatelists.conf

elif [ ! -f \"\${gravityDBfile}\" ]; then
    echo \"Lists update skipped - gravity database not found\"
else
    /usr/bin/php /usr/local/sbin/pihole-updatelists --config=/etc/pihole-updatelists/pihole-updatelists.conf --no-gravity --no-reload \${SCRIPT_ARGS} > /var/log/pihole-updatelists-onboot.log

If I keep the etc map folders on the host (which has an existing gravity database), wipe the existing the pihole container, and start the docker again, than it runs as expected.

Attaching to pihole
pihole  | s6-rc: info: service s6rc-oneshot-runner: starting
pihole  | s6-rc: info: service s6rc-oneshot-runner successfully started
pihole  | s6-rc: info: service fix-attrs: starting
pihole  | s6-rc: info: service fix-attrs successfully started
pihole  | s6-rc: info: service legacy-cont-init: starting
pihole  | s6-rc: info: service legacy-cont-init successfully started
pihole  | s6-rc: info: service cron: starting
pihole  | s6-rc: info: service cron successfully started
pihole  | s6-rc: info: service _uid-gid-changer: starting
pihole  | s6-rc: info: service _uid-gid-changer successfully started
pihole  | s6-rc: info: service _startup: starting
pihole  | FTLCONF_REPLY_ADDR4 is deprecated. Converting to FTLCONF_LOCAL_IPV4
pihole  |   [i] Starting docker specific checks & setup for docker pihole/pihole
pihole  |   [i] Setting capabilities on pihole-FTL where possible
pihole  |   [i] Applying the following caps to pihole-FTL:
pihole  |         * CAP_CHOWN
pihole  |         * CAP_NET_BIND_SERVICE
pihole  |         * CAP_NET_RAW
pihole  |   [i] Ensuring basic configuration by re-running select functions from basic-install.sh
pihole  |
pihole  |   [i] Installing configs from /etc/.pihole...
pihole  |   [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
  [✓] Installed /etc/dnsmasq.d/01-pihole.conf
  [✓] Installed /etc/dnsmasq.d/06-rfc6761.conf
pihole  |
  [✓] Installing latest logrotate script
pihole  |   [i] Creating empty /etc/pihole/setupVars.conf file.
pihole  |   [i] Assigning password defined by Environment Variable
pihole  |   [✓] New password set
pihole  |   [i] Setting Web Theme based on WEBTHEME variable, using value lcars
pihole  |   [i] Added ENV to php:
pihole  |                       "TZ" => "America/Los_Angeles",
pihole  |                       "PIHOLE_DOCKER_TAG" => "2022.09.1",
pihole  |                       "PHP_ERROR_LOG" => "/var/log/lighttpd/error-pihole.log",
pihole  |                       "CORS_HOSTS" => "",
pihole  |                       "VIRTUAL_HOST" => "192.168.2.177",
pihole  |   [i] Using IPv4 and IPv6
pihole  |   [i] setup_blocklists now setting default blocklists up:
pihole  |   [i] TIP: Use a docker volume for /etc/pihole/adlists.list if you want to customize for first boot
pihole  |   [i] Blocklists (/etc/pihole/adlists.list) now set to:
pihole  | https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
pihole  |   [i] Setting DNS servers based on PIHOLE_DNS_ variable
pihole  |   [i] Applying pihole-FTL.conf setting LOCAL_IPV4=192.168.2.177
pihole  |   [i] Applying pihole-FTL.conf setting REPLY_ADDR4=192.168.2.177
pihole  |   [i] FTL binding to default interface: eth0
pihole  |   [i] Enabling Query Logging
pihole  |   [i] Testing lighttpd config: Syntax OK
pihole  |   [i] All config checks passed, cleared for startup ...
pihole  |   [i] Docker start setup complete
pihole  |
pihole  |   Pi-hole version is v5.12 (Latest: v5.12)
pihole  |   AdminLTE version is v5.14.1 (Latest: v5.14.1)
pihole  |   FTL version is v5.17 (Latest: v5.17)
pihole  |   Container tag is: 2022.09.1
pihole  |
pihole  |   [i] pihole-FTL (no-daemon) will be started as pihole
pihole  |
pihole  | s6-rc: info: service _startup successfully started
pihole  | s6-rc: info: service pihole-FTL: starting
pihole  | s6-rc: info: service pihole-FTL successfully started
pihole  | s6-rc: info: service _updatelistsonboot: starting
pihole  | s6-rc: info: service lighttpd: starting
pihole  | s6-rc: info: service _updatelistsonboot successfully started
pihole  | s6-rc: info: service lighttpd successfully started
pihole  | s6-rc: info: service _gravityonboot: starting
pihole  | '/etc/pihole-updatelists.conf' -> '/etc/pihole-updatelists/pihole-updatelists.conf'
pihole  | s6-rc: info: service _gravityonboot successfully started
pihole  | s6-rc: info: service legacy-services: starting
pihole  |   Checking if custom gravity.db is set in /etc/pihole/pihole-FTL.conf
pihole  | ownership of '/etc/pihole-updatelists/pihole-updatelists.conf' retained as root:root
pihole  | mode of '/etc/pihole-updatelists/pihole-updatelists.conf' retained as 0644 (rw-r--r--)

pihole | Lists update skipped - gravity database not found

pihole  | services-up: info: copying legacy longrun pihole-dot-doh (no readiness notification)
pihole  | s6-rc: info: service legacy-services successfully started

pihole | [i] Creating new gravity database

pihole  | Starting stubby
pihole  | [16:36:29.029579] STUBBY: Stubby version: Stubby 0.3.0
pihole  | [16:36:29.035938] STUBBY: Read config from file /config/stubby.yml
pihole  | Starting cloudflared
pihole  | 2022-09-03T16:36:29Z INF Version 2022.8.4
pihole  | 2022-09-03T16:36:29Z INF GOOS: linux, GOVersion: go1.18.5, GoArch: arm64
pihole  | 2022-09-03T16:36:29Z INF Settings: map[config:/config/cloudflared.yml proxy-dns:true proxy-dns-address:127.1.1.1 proxy-dns-port:5153 proxy-dns-upstream:[https://family.cloudflare-dns.com/dns-query]]
pihole  | 2022-09-03T16:36:29Z INF Adding DNS upstream url=https://family.cloudflare-dns.com/dns-query
pihole  | 2022-09-03T16:36:29Z INF Starting DNS over HTTPS proxy server address=dns://127.1.1.1:5153
pihole  | 2022-09-03T16:36:29Z INF cloudflared will not automatically update if installed by a package manager.
pihole  |   [i] Migrating content of /etc/pihole/adlists.list into new database
pihole  |   [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range
pihole  |
  [✓] Preparing new gravity database
pihole  |   [i] Using libz compression
pihole  |
pihole  |   [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful
pihole  |   [i] Analyzed 136768 domains
pihole  |
  [✓] Creating new gravity databases
  [✓] Storing downloaded domains in new gravity database
  [✓] Building tree
  [✓] Swapping databases
pihole  |   [✓] The old database remains available.
pihole  |   [i] Number of gravity domains: 136768 (136768 unique domains)
pihole  |   [i] Number of exact blacklisted domains: 0
pihole  |   [i] Number of regex blacklist filters: 0
pihole  |   [i] Number of exact whitelisted domains: 0
pihole  |   [i] Number of regex whitelist filters: 0
  [✓] Cleaning up stray matter
pihole  |
pihole  |   [✓] FTL is listening on port 53
pihole  |      [✓] UDP (IPv4)
pihole  |      [✓] TCP (IPv4)
pihole  |      [✓] UDP (IPv6)
pihole  |      [✓] TCP (IPv6)
pihole  |
pihole  |   [i] Pi-hole blocking will be enabled
pihole  |   [i] Enabling blocking
jacklul commented 1 year ago

Unfortunately I don't see how to accomplish this - you simply gotta restart the image currently. Gravity on boot depends on pihole-updatelists-onboot, if I swap them around (to fix this issue) you will have to run gravity update TWICE each time image starts.

I get the "Lists update skipped - gravity database not found" message, which means updatelists is not using /etc/pihole-updatelists/pihole-updatelists.conf file, still using /etc/pihole-updatelists.conf

It doesn't even run in this case, it's not using either of them

I pushed a fix for something else though - forgot to correct the command line for the crontab - it still used the old config

mwatz1234 commented 1 year ago

I noticed I only see this issue, when other containters (In my instance Home-Assistant) are using network_mode: host. On my other pi, which none of the conatiners use host for the network mode has 0 issues.

If i add network_mode: host to docker compose for my custom docker, than I no longer see an issue, even with the first startup it pulls adlists as defined.

jacklul commented 1 year ago

Yeah well I have no idea what's that, I won't be much of a help here