search

pi-hole / docker-pi-hole

Pi-hole in a docker container
https://pi-hole.net
Other
8.63k stars 1.14k forks source link

upstream DNS set as env are not set properly after regeneration/restart of container #1275

Closed bitdruid closed 1 year ago

bitdruid commented 1 year ago

This is a: BUG

Details

The pihole container is set up, envs for upstream DNS were set according to readme. After the container starts/restarts or is regenerated the env seems to be set up "visually" - i can see my settings in the pihole settings-interface (for me e.g. 9.9.9.9 and 1.1.1.1) but pihole throws following warning until i click on ,,save". Until then no dns-resolution by pihole:

DNSMASQ_WARN Warning in dnsmasq core: no upstream servers configured

Related Issues

How to reproduce the issue

  1. Environment data

    • Operating System: Rasbian bullseye
    • Hardware: Raspi 4B 2GB
    • Kernel Architecture: ARMv8, aarch64, Kernel 5.15.84-v8+
    • Docker Install Info and version:
    • Software source: docker.io official latest 20.10.22
    • Supplimentary Software: portainer 2.16.2 community
    • Hardware architecture: ARMv8

  2. docker-compose.yml contents, docker run shell command, or paste a screenshot of any UI based configuration of containers here

version: "3.6" services: pihole: cap_drop:

  • "AUDIT_CONTROL"
  • "BLOCK_SUSPEND"
  • "DAC_READ_SEARCH"
  • "IPC_LOCK"
  • "IPC_OWNER"
  • "LEASE"
  • "LINUX_IMMUTABLE"
  • "MAC_ADMIN"
  • "MAC_OVERRIDE"
  • "NET_ADMIN"
  • "NET_BROADCAST"
  • "SYSLOG"
  • "SYS_ADMIN"
  • "SYS_BOOT"
  • "SYS_MODULE"
  • "SYS_NICE"
  • "SYS_PACCT"
  • "SYS_PTRACE"
  • "SYS_RAWIO"
  • "SYS_RESOURCE"
  • "SYS_TIME"
  • "SYS_TTY_CONFIG"
  • "WAKE_ALARM" container_name: "pihole" environment:
  • "PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
  • "phpver=php"
  • "PHP_ENV_CONFIG=/etc/lighttpd/conf-enabled/15-fastcgi-php.conf"
  • "PHP_ERROR_LOG=/var/log/lighttpd/error-pihole.log"
  • "IPv6=False"
  • "S6_KEEP_ENV=1"
  • "S6_BEHAVIOUR_IF_STAGE2_FAILS=2"
  • "S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0"
  • "FTLCONF_LOCAL_IPV4=127.0.0.1"
  • "FTL_CMD=no-daemon"
  • "DNSMASQ_USER=pihole"
  • "TZ=Europe/Berlin"
  • "DNSMASQ_LISTENING=all"
  • "PIHOLEDNS=1.1.1.1;9.9.9.9" hostname: "pihole" image: "docker.io/pihole/pihole:latest" network_mode: "bridge" ports:
  • "53:53/tcp"
  • "53:53/udp"
  • "50001:80/tcp" restart: "unless-stopped" volumes:
  • "pihole_dnsmasq:/etc/dnsmasq.d"
  • "pihole_etc:/etc/pihole"
  1. any additional info to help reproduce

These common fixes didn't work for my issue

If the above debugging / fixes revealed any new information note it here. Add any other debugging steps you've taken or theories on root cause that may help.

Debug-Log in addition (could not provide a token because no upstreams - was before clicking ,,save")

pihole_diagnostic.txt

Screenshot 2023-01-16 155617 Screenshot 2023-01-16 155252

PromoFaux commented 1 year ago

This is the third such case I've seen of this - but still am unable to reproduce.

@rdwebdesign saw it happen on one of his containers, and now he is unable to reproduce it again

PromoFaux commented 1 year ago

My only thought is maybe some sort of permissions issue on the volume.

I was seeing some issues on one my install in the error log - but I cleared down and recreated the volumes and the errors have gone away (ignore the warnings about the config-key, those will be sorted in the next release)

image

rdwebdesign commented 1 year ago

I'm not sure if I can help, because I don't know what happened and I can't replicate the issue now.

I saw the same warning message when I updated one of my Pi-hole containers. The other one, on a different Raspberry Pi was updated without issues.

Details: After updating the image (from 2022.12.1 to 2023.01), the error appeared.

Using web interface Settings Page (DNS tab), I saved the same config (without changes, I just clicked on the Save button) and the warning disappeared, but restarting the image (or rebooting the machine) the warning reappeared every time.

I was using bind volumes. I didn't checked the permissions when I saw the error.

To compare the behavior, I decided to go back to the previous image (2022.12.1). It worked without warnings. When I change back to 2023.01 again, the warning never appeared again. I can't replicate the previous state.

bitdruid commented 1 year ago

My only thought is maybe some sort of permissions issue on the volume.

I was seeing some issues on one my install in the error log - but I cleared down and recreated the volumes and the errors have gone away (ignore the warnings about the config-key, those will be sorted in the next release)

image

thanks for your reply! if it were necessary I could provide some logs. I rebuilt the container without persistent storage (so I doubt permission problems in the volumes?). the error remained... i took a snapshot of the system state and reverted back to my non-docker configuration. i will try to rebuild the container from scratch tomorrow

Bortinio commented 1 year ago

I had the same issue with 2023.01 and reverted back to 2022.12.1. Although set by the env variable PIHOLEDNS the server was not written to /etc/dnsmasq.d/01-pihole.conf and as such queries did not get answered. I also noticed that the setting interface= was not set by 2023.01. With 2022.12.1 it was set to interface=eth0, although I did not set it by an env variable in the docker compose.

djrm05 commented 1 year ago

I had the same issue with 2023.01 and reverted back to 2022.12.1. Although set by the env variable PIHOLEDNS the server was not written to /etc/dnsmasq.d/01-pihole.conf and as such queries did not get answered. I also noticed that the setting interface= was not set by 2023.01. With 2022.12.1 it was set to interface=eth0, although I did not set it by an env variable in the docker compose.

I have seen the same in 2 different containers. Just going to settings-DNS and saving fix the issue, but it appears if container is restarted.

rdwebdesign commented 1 year ago

@djrm05

How are you storing the config files? using bind volumes?

Can you please post the output of ls -ls on the directory holding the dnsmasq.d config?

Szesi111 commented 1 year ago

I have the same issue.

volumes /home/homeassistant/pihole/etc-dnsmasq.d /etc/dnsmasq.d /home/homeassistant/pihole/etc-pihole /etc/pihole

After container restart (without save):

ls -l /home/homeassistant/pihole/etc-dnsmasq.d -rw-r--r-- 1 root root 1334 jan 17 10.45 01-pihole.conf -rw-r--r-- 1 root root 2190 jan 17 10.45 06-rfc6761.conf

cat 01-pihole.conf addn-hosts=/etc/pihole/local.list addn-hosts=/etc/pihole/custom.list domain-needed localise-queries bogus-priv no-resolv log-queries log-facility=/var/log/pihole/pihole.log log-async interface= cache-size=10000

After save: ls -l /home/homeassistant/pihole/etc-dnsmasq.d -rw-r--r-- 1 root root 1455 jan 17 10.46 01-pihole.conf -rw-r--r-- 1 root root 2190 jan 17 10.45 06-rfc6761.conf

cat 01-pihole.conf addn-hosts=/etc/pihole/local.list addn-hosts=/etc/pihole/custom.list localise-queries no-resolv log-queries log-facility=/var/log/pihole/pihole.log log-async cache-size=10000 server=208.67.222.222 server=208.67.220.220 interface=eth0 rev-server=192.168.199.0/24,192.168.199.1 server=/home.local/192.168.199.1 server=//192.168.199.1

djrm05 commented 1 year ago

I'm using standard volumes (no bind).

@.***:/var/lib/docker/volumes/pihole_dnsmasqd/_data# ls -ls total 16 4 -rw-r--r-- 1 root root 1352 ene 17 11:14 01-pihole.conf 4 -rwxrwxrwx 1 root root 35 ene 30 2022 02-custom.conf 4 -rw-r--r-- 1 root root 2190 ene 17 11:14 06-rfc6761.conf 4 -rw-r--r-- 1 root root 21 feb 5 2022 99-edns.conf

El lun, 16 ene 2023 a las 22:58, RD WebDesign @.***>) escribió:

@djrm05 https://github.com/djrm05

How are you storing the config files? using bind volumes?

Can you please post the output of ls -ls on the directory holding the dnsmasq.d config?

— Reply to this email directly, view it on GitHub https://github.com/pi-hole/docker-pi-hole/issues/1275#issuecomment-1384606913, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIEGEO42M26VXQP7GOMWLCLWSXABJANCNFSM6AAAAAAT43YVFU . You are receiving this because you were mentioned.Message ID: @.***>

kwlorenz commented 1 year ago

Same problem here with latest pihole, had to go back to 2022.12.1, problem gone ...

Recte commented 1 year ago

Same problem on Odroid running 'Armbian 22.11.2 Bullseye with Linux 5.19.17', where the previous version (pihole/pihole:2022.12.1) works flawless.

MisterBearBear commented 1 year ago

Having the exact same problem after upgraded to 2023.01. Had the same observations like others:

  1. The "no upstream servers configured" error happened every time the container recreated or restarted. Pi-hole will run but it won't handle any DNS queries (look at the blank period in the screenshot below)

image

  1. Go to Settings and the DNS tab, my previous upstream DNS server configurations was still there even though the "no upstream servers configured" error occurred.

image

  1. Simply click the Save button on the Upstream DNS Server page will resolve the issue. But the issue will come back after the container recreated or restarted.

Hope this helps.

Hardware and Software Info:

GitHubAppleDreamer commented 1 year ago

I have the same problem. MisterBearBear describes the error exactly as it appears on my system. I use Pi-Hole in Docker on a Synology. I am now back to 2022.12.1 and have no more problems.

PromoFaux commented 1 year ago

I have just seen this in somebodies logs on Reddit:

grep: /etc/lighttpd/conf-enabled/15-fastcgi-php.conf: No such file or directory

We changed the default value of an environment variable in the Dockerfile (actually - I don't really know why this is an environment variable thinking about it....) But it explains why it's failing for some folk and not others...

The PHP_ENV_CONFIG variable used to have the value /etc/lighttpd/conf-enabled/15-fastcgi-php.conf, it now has /etc/lighttpd/conf-enabled/15-pihole-admin.conf... anyone seeing this issue is likely retaining this old value.

Are all of you completely destoying your containers before building them again, per the upgrade section in the Readme?

But to go back to wahat I said - I don't really see why this is an environment variable anyway - the value doesn't need to be configurable - I'll fix it

MisterBearBear commented 1 year ago

I "recreated" the container in Portainer to pull the latest image so it might keep the old ENV variables.

Now I have the PHP_ENV_CONFIG deleted and it works fine now. Thanks.

PromoFaux commented 1 year ago

Thanks, that confirms the issue - Can somebody that is also seeing this issue please try the :dev tag before I push it to a new version?

djrm05 commented 1 year ago

I recreated the container in portainer like @MisterBearBear did. I have just tested :dev tag without removing the PHP_ENV_CONFIG in one of the containers that had the issue and is working perfectly.

PromoFaux commented 1 year ago

2023.01.1 building as we speak https://github.com/pi-hole/docker-pi-hole/actions/runs/3950828292

MisterBearBear commented 1 year ago

2023.01.1 building as we speak https://github.com/pi-hole/docker-pi-hole/actions/runs/3950828292

Thanks for the help and the great work!

djrm05 commented 1 year ago

2023.01.1 building as we speak https://github.com/pi-hole/docker-pi-hole/actions/runs/3950828292

thanks!! :)

MisterBearBear commented 1 year ago

Just updated to 2023.01.1. It is working without issues. Thanks again.

Bortinio commented 1 year ago

I can confirm that 2023.01.1 also fixed it for me now. Thanks a lot!

PromoFaux commented 1 year ago

Copy paste error in 2023.01.1....2023.01.2 building now 🫥

MisterBearBear commented 1 year ago

image

Typo in the version number?

rdwebdesign commented 1 year ago

Yeah... we will need to re-tag it.

Thanks.

PromoFaux commented 1 year ago

I made it 18 days into the year before I wrote the year down wrong.

Szesi111 commented 1 year ago

2023.01.1 and 2023.01.2 also fixed it. Thanks!

ppuschmann commented 1 year ago

🤔 somehow I still have this issue, running pi-hole in k8s.

If I use the env-var PIHOLE_DNS_, the upstream-servers aren't set.

If I use the (old) env-vars like PIHOLE_DNS_1, PIHOLE_DNS_2 and so on, pi-hole will come up ~running just fine~, with all upstream servers configured.

Version: 2023.03.1

edit: pi-hole is coming up, but DNS-resolution is broken, until you save the DNS-servers again. 😕

I'll create some proper debug-log and might start a new issue.

dschaper commented 1 year ago

If I use the env-var PIHOLE_DNS_, the upstream-servers aren't set.

If I use the (old) env-vars like PIHOLE_DNS_1, PIHOLE_DNS_2 and so on, pi-hole will come up ~running just fine~, with all upstream servers configured.

Can you please post the exact configuration lines/yaml that you are using?

sebastianliebscher commented 1 year ago

I have never had this error before, but now with release of 2023.11.0 I get the same issue on my deployment.

After a restart or stop+start of the Docker stack, I get the following error:

s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service cron: starting
s6-rc: info: service cron successfully started
s6-rc: info: service _uid-gid-changer: starting
s6-rc: info: service _uid-gid-changer successfully started
s6-rc: info: service _startup: starting
  [i] Starting docker specific checks & setup for docker pihole/pihole
  [i] Setting capabilities on pihole-FTL where possible
  [i] Applying the following caps to pihole-FTL:
        * CAP_CHOWN
        * CAP_NET_BIND_SERVICE
        * CAP_NET_RAW
  [i] Ensuring basic configuration by re-running select functions from basic-install.sh
  [i] Installing configs from /etc/.pihole...
  [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
  [i] Installing /etc/dnsmasq.d/01-pihole.conf...
  [✓] Installed /etc/dnsmasq.d/01-pihole.conf
  [i] Installing /etc/.pihole/advanced/06-rfc6761.conf...
  [✓] Installed /etc/dnsmasq.d/06-rfc6761.conf
s6-rc: info: service _startup successfully started
s6-rc: info: service pihole-FTL: starting
s6-rc: info: service pihole-FTL successfully started
s6-rc: info: service lighttpd: starting
s6-rc: info: service lighttpd successfully started
s6-rc: info: service _postFTL: starting
s6-rc: info: service _postFTL successfully started
s6-rc: info: service legacy-services: starting
  Checking if custom gravity.db is set in /etc/pihole/pihole-FTL.conf
s6-rc: info: service legacy-services successfully started
  [✗] DNS resolution is currently unavailable
  [i] Time until retry: 120

Even if the correct setting is displayed in the UI, one must first manually save to apply the setting.

upstream-ip

Also, note how I specified upstream DNS hostnames (Docker service names are supported) but in the UI their IP is shown. Maybe it is related to the issue, maybe not.

Steps to reproduce:

  1. (have a successfully running docker-pi-hole prior to 2023.11.0)
  2. upgrade to 2023.11.0: docker-compose.yml
services:
  pihole:
    image: pihole/pihole:2023.11.0
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "8888:80/tcp"
    environment:
      TZ: 'Europe/Berlin'
      PIHOLE_DNS_: hostname-1;hostname-2
      DNSSEC: true
    volumes:
      - '/some/dir/:/etc/pihole/'
      - '/some/dir2/:/etc/dnsmasq.d/'
    restart: unless-stopped
  1. restart the stack
  2. get the error
PromoFaux commented 1 year ago

Maybe it is related to the issue, maybe not.

The docker hostnames are converted to IP addresses on container start (this may be different with the v6 containers, I don't remember off hand)

What version of the container were you running before upgrade to 2023.11.0? There have been no docker specific changes for a while

sebastianliebscher commented 1 year ago

Thanks for your reply.

The docker hostnames are converted to IP addresses on container start (this may be different with the v6 containers, I don't remember off hand)

Alright, so it is intended behavior and shouldn't be related to the issue.

There have been no docker specific changes for a while

2023.10.0. My config hasn't changed in the last years, but I always try to use the latest stable version.

Edit: Could this be related to the issue?

[i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
PromoFaux commented 1 year ago

latest stable version.

Just as a point of clarification - the version numbers are just date based, not semantic. Any release is considered "stable" (until proven otherwise 😉 🫣)

What are the contents of /etc/dnsmasq.d/01-pihole.conf?

And no, that's normal - have just started up my 2022.11.0 container:

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service cron: starting
s6-rc: info: service cron successfully started
s6-rc: info: service _uid-gid-changer: starting
s6-rc: info: service _uid-gid-changer successfully started
s6-rc: info: service _startup: starting
  [i] Starting docker specific checks & setup for docker pihole/pihole
  [i] Setting capabilities on pihole-FTL where possible
  [i] Applying the following caps to pihole-FTL:
        * CAP_CHOWN
        * CAP_NET_BIND_SERVICE
        * CAP_NET_RAW
        * CAP_NET_ADMIN
  [i] Ensuring basic configuration by re-running select functions from basic-install.sh
  [i] Installing configs from /etc/.pihole...
  [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
  [i] Installing /etc/dnsmasq.d/01-pihole.conf...
  [✓] Installed /etc/dnsmasq.d/01-pihole.conf
  [i] Installing /etc/.pihole/advanced/06-rfc6761.conf...
  [✓] Installed /etc/dnsmasq.d/06-rfc6761.conf
  [i] Installing latest logrotate script...
    [i] Existing logrotate file found. No changes made.
  [i] Assigning password defined by Environment Variable
  [✓] New password set
  [i] Added ENV to php:
                    "TZ" => "europe/London",
                    "PIHOLE_DOCKER_TAG" => "",
                    "PHP_ERROR_LOG" => "/var/log/lighttpd/error-pihole.log",
                    "CORS_HOSTS" => "",
                    "VIRTUAL_HOST" => "docker-pihole-pi4.lan",

I think we might be seeing a slightly different issue to the OP here, can you please create a new issue and we can troubleshoot there if needs be - just to stop the notification noise for everyone else in this thread... :)