search

pi-hole / docker-pi-hole

Pi-hole in a docker container
https://pi-hole.net
Other
8.56k stars 1.13k forks source link

Stopped blocking after upgrade [arm64/aarch64 raspbian] #587

Closed Mellbourn closed 3 years ago

Mellbourn commented 4 years ago

After upgrading to v4.4 my pi hole stopped blocking ads.

This is a...

Description

I have been using docker-pi-hole without issue for months. Recently I did an upgrade (using docker-compose pull && docker-compose up -d, which upgraded to v4.4 After this upgrade, the pi-hole seems to be up and running, the web interface is working, but after running for several days:

I'm a bit surprised at the FTL Version, it says vDev (development, vDev-7dea0f0) Screenshot 2020-02-28 at 14 58 30

This is starting to sound like I have gotten a 5.0 beta version of the FTL, where you are supposed to have a database rather than a list. But I have not done anything to get a beta, I have just done a normal docker-compose pull.

Tested doing pihole checkout ftl master but that did not help. It didn't even change version number displayed in GUI, so I'm not sure it worked.

Tested changing docker-compose.yml so that instead of getting pihole/pihole:latest it would get pihole/pihole:release-v4.3.4, and doing docker-compose pull but that didn't really help, it did downgrade the Pi-hole version, but not the FTL version.

Expected Behavior

Ads blocked.

Actual Behavior

No ads blocked.

Possible Fix

Somehow getting the correct FTL version, maybe?

Steps to Reproduce and debugging done

  1. docker-compose pull
  2. docker-compose up -d

Debug steps I have tried

These things did not help:

  1. I restarted the pi-hole.
  2. I have restarted the docker host.
  3. I have done a update gravity through the web gui, it reported no problems.
  4. I have checked the contents of /etc/pihole, and it looks normal. The list.*.domains files are there and are normal size.
  5. I have flushed the logs

Debug log: https://tricorder.pi-hole.net/y9e5xygthd

I can't think of any workaround, so pi-hole has completely stopped working for me.

Context and extra information

Your Environment

ryand11 commented 4 years ago

I am having the same issue on the same build

loguerto commented 4 years ago

Same issue here.

nathang21 commented 4 years ago

Has anyone successfully upgraded without this issue?

I'm a little concerned about upgrading.

On Sun, Mar 1, 2020 at 10:32 AM loguerto notifications@github.com wrote:

Same issue here.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pi-hole/docker-pi-hole/issues/587?email_source=notifications&email_token=AB4CUJX5CL5MSZKV2ZAM7DTRFKS2LA5CNFSM4K5QOG3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENNG42Q#issuecomment-593129066, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4CUJRKFVSMEFDSJNVBCSLRFKS2LANCNFSM4K5QOG3A .

PromoFaux commented 4 years ago

I have just updated my container running 4.3.2 to use the :v4.4 tagged image (taken me a while to get around to it!), and am not seeing any issues.

image

:latest should be at 4.4, too.

I guess I will leave it running a few days to see if any of this is reproducible.

You definitely should not be seeing that version of FTL in this container.

What does your compose file look like?

RadiusK commented 4 years ago

Same issue here ever since Watchtower automatically upgraded my pihole, was using latest tag at the time. Tried downgrading to release-v4.3.4 just like @Mellbourn , without luck. Directly using v4.4 doesn't fix it either. Have played around by removing some of the persisted data in /etc/pihole/, and even started over by removing all data+container and starting a fresh container. Not sure what determines the version of FTL, but it doesn't seem to be affected by the different images or even presence of pre-existing data or custom block & blacklists.

My compose file is pretty straight forward. Thing worth mentioning might be that I'm using host as network-mode and using pihole for DHCP. It worked perfectly for a couple of days (only got into last week), but now it interestingly shows -2 items on my blocklist and doesn't block anything.

pihole -v output

Pi-hole version is v4.4 (Latest: v4.4)
AdminLTE version is v4.3.3 (Latest: v4.3.3)
FTL version is vDev-7dea0f0 (Latest: v4.3.1)

docker-compose.yml

services:
  pihole:
    container_name: pihole
    image: pihole/pihole:v4.4
    network_mode: host
    environment:
      - TZ=UTC
      - WEBPASSWORD=<my_password>
      - ServerIP=<my_host_ip_address>
    volumes:
      - './etc-pihole/:/etc/pihole/'
      - './etc-dnsmasq.d/:/etc/dnsmasq.d/'
    dns:
      - 127.0.0.1
      - 1.1.1.1
    cap_add:
      - NET_ADMIN
    restart: unless-stopped

Don't see anything strange in the logs when starting the container. The number of unique trapped domains is a lot higher than minus two though :P 

pihole    |
  [✓] Consolidating blocklists
  [✓] Extracting domains from blocklists
pihole    |   [i] Number of domains being pulled in by gravity: 147707
  [✓] Removing duplicate domains
pihole    |   [i] Number of unique domains trapped in the Event Horizon: 125377
Mellbourn commented 4 years ago

This is my docker-compose.yml

version: "3.6"

 # More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
 services:
   pihole:
     container_name: pihole
     image: pihole/pihole:latest
     ports:
       - "53:53/tcp"
       - "53:53/udp"
       - "67:67/udp"
       - "80:80/tcp"
       - "443:443/tcp"
     environment:
       TZ: 'Europe/Stockholm'
       WEBPASSWORD: 'the password'
       # this is still needed to get pi.hole to work
       ServerIP: '192.168.0.9'
     # this makes the name of the docker image not be a random hash. This name is recognized only in the docker host environment
     hostname: 'pi-hole-in-docker'
     # Volumes store your data between container upgrades
     volumes:
       - './pi-hole/etc-pihole/:/etc/pihole/'
       - './pi-hole/etc-dnsmasq.d/:/etc/dnsmasq.d/'
     # these are supposedly needed, otherwise you will get 127.0.0.11 problems https://github.com/pi-hole/docker-pi-hole/issues/410#issuecomment-460515706 - s till doesn't work
     dns:
       - 127.0.0.1
       - 1.1.1.1
     # Recommended but not required (DHCP needs NET_ADMIN)
     #   https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
     cap_add:
       - NET_ADMIN
     restart: unless-stopped
RadiusK commented 4 years ago

I just downgraded even further back to pihole/pihole:4.3.2-1 (which seems to been have a stable build for multiple months), and my FTL version is OK again! Blocklist is showing correct number and the blocked domains appear to be blocked once again 👍 .

Pi-hole version is v4.3.2 (Latest: v4.4)
AdminLTE version is v4.3.2 (Latest: v4.3.3)
FTL version is v4.3.1 (Latest: v4.3.1)

Issue seems to have been introduced between 4.3.2-1 and 4.3.4.

lightswitch05 commented 4 years ago

@Mellbourn if you run docker images pihole/pihole:latest, what is the value for your IMAGE ID?

PromoFaux commented 4 years ago

Could I also please get some debug tokens from people that currently have issues?

diginc commented 4 years ago

Like lightswitch said, please check the image sha closely and make sure you run a fresh docker pull pihole/pihole to see if anything new comes through.

Latest Image ID should be:

$ docker images --no-trunc pihole/pihole:latest
REPOSITORY          TAG                 IMAGE ID                                                                  CREATED             SIZE
pihole/pihole       latest              sha256:694e8aa9d588114780edd8455a7f7b3a38c5a2ba6fd159ff579ef85e3754b995   6 days ago          289MB

And confirm your pihole container uses that same sha with (inspect container name):

$ docker inspect pihole | grep Image
        "Image": "sha256:694e8aa9d588114780edd8455a7f7b3a38c5a2ba6fd159ff579ef85e3754b995",
            "Image": "pihole/pihole",

I've also been running this image at home since it was released and checking it every day or two for any issue (non found yet).

Mellbourn commented 4 years ago

Debug token is: https://tricorder.pi-hole.net/vunyrkpic6

> docker images --no-trunc pihole/pihole:latest
REPOSITORY          TAG                 IMAGE ID                                                                  CREATED             SIZE
pihole/pihole       latest              sha256:623d099087d50c273380002b221f7c5e6691fb603bafbea949bc1e72857d1ed0   6 days ago          324MB

> docker inspect pihole| grep Image
        "Image": "sha256:623d099087d50c273380002b221f7c5e6691fb603bafbea949bc1e72857d1ed0",
            "Image": "pihole/pihole:latest",

So my image id is not the same as yours. Running a docker pull pihole/pihole does not pull anything new: Image is up to date

Maybe I should point out that my host system is running a 64 bit kernel, aarch64 (More information at https://www.raspberrypi.org/forums/viewtopic.php?t=250730)

I can also confirm what @RadiusK found: If I downgrade to pihole/pihole:4.3.2-1 everything starts working again.

dschaper commented 4 years ago

What does your docker info show for Kernel/Os/Arch etc...?

dschaper commented 4 years ago

https://hub.docker.com/layers/pihole/pihole/4.3.2-1_aarch64/images/sha256-9572225c3713dc4cc4d079a28788f7e69f14efa88812356b5aa9e058fa6dcae1?context=repo

Mellbourn commented 4 years ago

@dschaper :

> docker info                                                                                               [v5-beta|v] pi@raspberrypi4docker 23:46:50
Client:
 Debug Mode: false

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 4
 Server Version: 19.03.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 35bd7a5f69c13e1563af8a93431411cd9ecf5021
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.97-v8+
 Operating System: Raspbian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 3.718GiB
 Name: nameofmydockerhost
 ID: GHLE:KKE6:UXZK:PQM4:KU3Q:MKCL:4D7B:XDYQ:4F75:YTX7:2OYF:XW2X
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
PromoFaux commented 4 years ago

What I can't quite wrap my head around is how the downloaded FTL binary is on that commit... very bizarre!

diginc commented 4 years ago

Hmmm - checking latest arm64 out and get the right version

docker run -it --rm --entrypoint pihole pihole/pihole:v4.4-arm64 -v
Digest: sha256:68049b449be38b67506758a30898778fd77052a48067b722ac14d929eed36a0a
Status: Downloaded newer image for pihole/pihole:v4.4-arm64
  Pi-hole version is v4.4 (Latest: v4.4)
  AdminLTE version is v4.3.3 (Latest: v4.3.3)
  FTL version is v4.3.1 (Latest: v4.3.1)
diginc commented 4 years ago

So my image id is not the same as yours.

Me overlooking arm64 / aarch64 explains that, but the above image hash in my previous post still doesn't match. Might be because I'm not really arm64 but instead am doing an amd64 docker qemu run, unsure.

I went through a few pages of published tags @ https://hub.docker.com/r/pihole/pihole/tags and couldn't find 623d099087 or the full sha256 anywhere else I looked so either. This is quite the mystery.

Running a docker pull pihole/pihole does not pull anything new: Image is up to date

Lets try cleaning the existing image completely before pulling. Are you OK with cleaning all unused images? docker image prune should really clean stuff up completely but will hit ALL unused image layers. You have to stop your pihole container first to make it unused:

docker-compose stop pihole
docker-compose rm pihole
docker rmi pihole/pihole:latest
docker image prune 
docker pull pihole/pihole:latest
# For fun compare it to what :latest should auto-map to:
docker pull pihole/pihole:v4.4-amd64

docker-compose up -d pihole

Can you provide the sha256 hashes from both those pull commands?

diginc commented 4 years ago

On the off chance that doesn't work another thought that is a bit more far fetched comes to mind - the ~/.docker/ folder may have a manifests folder that is caching the wrong version. i had an issue with this folder for pushing images not updating to newer versions but never for pulling images...so I'm unsure if this is the same.

https://github.com/pi-hole/docker-pi-hole/blob/d880f69836bd50154f0cfc625e1a8ec36a0e5ab6/deploy_docker.sh#L65

so check if it exists before nuking it:

ls ~/.docker/
rm -rf ~/.docker/manifests/*

But along that path you might try just nuking everything docker more and more, like erasing all containers/image/volume storage by removing the default location for that (docker should re-create on next startup):

systemctl stop docker
rm -rf /var/lib/docker/
systemctl start docker
docker run -it --rm --entrypoint pihole pihole/pihole -v

Failing that a full reinstall of docker's apt package may help too.

Edit: sorry for so much stuff at once

Mellbourn commented 4 years ago

I did notice that I couldn't find my hash on hub.docker.com, but I couldn't find yours either.

I did these commands that you suggested:

They all succeeded (I also removed all older images of pihole/pihole). Then I ran the pull:

> docker pull pihole/pihole:latest
latest: Pulling from pihole/pihole
e6af97f21922: Pull complete
0c82380595cf: Pull complete
705457d3a0d0: Pull complete
7ffe8147f899: Pull complete
d176a5c6679d: Pull complete
1c9a6dc73bde: Pull complete
d9219e29d444: Pull complete
b2345054dd67: Pull complete
Digest: sha256:8c172c4cf344232a137202811b0c13a0542551076798d7635ef5368f40d1fe71
Status: Downloaded newer image for pihole/pihole:latest
docker.io/pihole/pihole:latest

> docker images --no-trunc pihole/pihole:latest
REPOSITORY          TAG                 IMAGE ID                                                                  CREATED             SIZE
pihole/pihole       latest              sha256:623d099087d50c273380002b221f7c5e6691fb603bafbea949bc1e72857d1ed0   6 days ago          324MB

So, I got the same image again. Running docker-compose up -d results in the same incorrectly behaving pihole as before.

Pulling the comparison image gets me another hash:

> docker pull pihole/pihole:v4.4-amd64
v4.4-amd64: Pulling from pihole/pihole
6b5ca2365e03: Pull complete
0f99a558f10d: Pull complete
8b3592d37814: Pull complete
b3d3bf189c24: Pull complete
c853c0207dda: Pull complete
fc12ecacd94f: Pull complete
35cccecafddc: Pull complete
ab29c1f5a464: Pull complete
Digest: sha256:6b0ba01187c2af7768ada839f9c670a57326b340e0b775a2b1e7de675269d93a
Status: Downloaded newer image for pihole/pihole:v4.4-amd64
docker.io/pihole/pihole:v4.4-amd64

> docker images --no-trunc pihole/pihole:v4.4-amd64
REPOSITORY          TAG                 IMAGE ID                                                                  CREATED             SIZE
pihole/pihole       v4.4-amd64          sha256:694e8aa9d588114780edd8455a7f7b3a38c5a2ba6fd159ff579ef85e3754b995   6 days ago          289MB
diginc commented 4 years ago

The big difference there is pihole/pihole:latest is a manifest image and v4.4-amd64 is a regular image. I'm still Thinking the arm64 install of docker has some manifest caching bug preventing those from updating. As a user of manifests I don't think ~/.docker/manifest exists but perhaps that cached data is somewhere else in /var/lib/docker that can be manually purged.

lightswitch05 commented 4 years ago

URL to get all images and their digests from Docker Hub:

If you grep the responses, your digest for pihole/pihole:v4.4-amd64 is found, but your digest for pihole/pihole:latest is not found

    {
      "creator": 8115145,
      "id": 88269528,
      "image_id": null,
      "images": [
        {
          "architecture": "amd64",
          "features": "",
          "variant": null,
          "digest": "sha256:6b0ba01187c2af7768ada839f9c670a57326b340e0b775a2b1e7de675269d93a",
          "os": "linux",
          "os_features": "",
          "os_version": null,
          "size": 129427093
        }
      ],
      "last_updated": "2020-02-26T18:40:14.804514Z",
      "last_updater": 8115145,
      "last_updater_username": "pralor",
      "name": "v4.4-amd64",
      "repository": 5756630,
      "full_size": 129427093,
      "v2": true
    },
diginc commented 4 years ago

Does the pihole/pihole:v4.4-amd64 image / sha256:6b0ba01187c2af7768ada839f9c670a57326b340e0b775a2b1e7de675269d93a image work better?

During my digging I also discovered it is possible to refer to images by sha256 - docker pull pihole/pihole@sha256:6b0ba01187c2af7768ada839f9c670a57326b340e0b775a2b1e7de675269d93a

The sha your latest points to errors out if you attempt to pull it.

Mellbourn commented 4 years ago

@diginc I haven't had time to do all the nuking you suggested, but I can mention that I don't have a ~/.docker folder.

Mellbourn commented 4 years ago

Trying to use pihole/pihole@sha256:6b0ba01187c2af7768ada839f9c670a57326b340e0b775a2b1e7de675269d93a in docker-compose.yml did not go well. docker-compose up -d goes into a loop of starting and restarting, the web gui is unreachable. Eventually the docker daemon crashed.

Mellbourn commented 4 years ago

@diginc now I've tried your suggestion:

systemctl stop docker
rm -rf /var/lib/docker/
systemctl start docker

And then doing docker-compose pull. I still get the same version as before.

Given I am not the only one having this problem, doesn't it seem unlikely that it is my docker installation or cache that has become corrupted? Isn't more plausible that the problem is elsewhere?

bugsyb commented 4 years ago

Hi Guys,

I've run into the very same problem - all was working fine and all of the sudden it stopped.

Looking into details, it might be something related to architecture regardless of how weird it sounds. The system in question is aarch64 somehow recognized for some images as arm64/v7, whilst it should be arm64/v8. This is due to the docker binary I suppose. It is Odroid N2 system on the HW side.

Back to this problem, I've had that weird situation of being on the latest one but running into the issue.

What fixed it for me was:

docker stop pihole ; docker container rm pihole; docker image rm pihole/pihole

docker pull --platform="arm64" pihole/pihole:latest

Somehow the arm/v7 is getting into Dev code as I've had also:

Pi-hole version is v4.4 (Latest: v4.4)
AdminLTE version is v4.3.3 (Latest: v4.3.3)
FTL version is vDev-7dea0f0 (Latest: v4.3.1)

After above I'm now on:

cat localbranches 
master master master
cat localversions 
v4.4-0-g9e49077 v4.3.3-0-g62f2ffc v4.3.1

Earlier it was showing:

cat localbranches 
master development
cat localversions
v4.4-0-g9e49077 vDev-7dea0f0

#within container - not sure what it was earlier though
dpkg --print-architecture
arm64

HTH.

Mellbourn commented 4 years ago

In my pi-hole the architecture prints as armhf outside the container, and armel inside:

> dpkg --print-architecture
armhf
> docker-compose exec pihole /bin/bash
root@pi-hole-in-docker:/# dpkg --print-architecture
armel

I tried using the --platform="arm64" parameter suggested by @bugsyb, but it seems to not be supported:

> docker pull --platform="arm64" pihole/pihole:latest
"--platform" is only supported on a Docker daemon with experimental features enabled
bugsyb commented 4 years ago

re @Mellbourn , I've updated daemon.json to below:

# cat daemon.json 
{
    "experimental" : true
}
Mellbourn commented 4 years ago

@bugsyb I saved the suggested content in /etc/docker/daemon.json and restarted the docker daemon. After that I could indeed use the --platform="arm64" flag without complaint. But it seems to have pulled the same digest as before:

> docker pull --platform="arm64" pihole/pihole:latest                                                                                  [v5-beta|+2] pi@raspberrypi4docker 19:47:10
latest: Pulling from pihole/pihole
5ed7afaa9adf: Pull complete
9a1f55b99947: Pull complete
722d4e0ba6ff: Pull complete
164ec22e03b6: Pull complete
f67b9381243f: Pull complete
e44673a7b448: Pull complete
1f47684a5487: Pull complete
f21e2c89ca46: Pull complete
Digest: sha256:8c172c4cf344232a137202811b0c13a0542551076798d7635ef5368f40d1fe71
Status: Downloaded newer image for pihole/pihole:latest
docker.io/pihole/pihole:latest

Starting that image makes the container go into a restart loop.

I also tried specifying a platform in the docker-compose.yml:

 platform: linux/arm64/v8

But that immediately resulted in an error:

ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services.pihole: 'platform'

That was apparently because I had specified version 3.6 in the docker-compose.yml so I downgraded it to version 2.4. This allowed docker-compose pull and docker-compose up -d to run, but again it resulted in a restart loop.

bugsyb commented 4 years ago

Check if you've removed previous image. If not, somehow it ignores new architecture. Try to re-run:

#docker stop pihole ; docker container rm pihole; docker image rm pihole/pihole
#docker pull --platform="arm64" pihole/pihole:latest

After that hopefully you'll get similar to below:

# docker run --platform="arm64" -it --rm --entrypoint sh  pihole/pihole:latest -c "pihole -v; dpkg --print-architecture"

  Pi-hole version is v4.4 (Latest: v4.4)
  AdminLTE version is v4.3.3 (Latest: v4.3.3)
  FTL version is v4.3.1 (Latest: v4.3.1)
arm64

# docker image inspect 505ad79f9e8a
[
    {
        "Id": "sha256:505ad79f9e8ad07ae4f5d3c470d77cf9c235f04785854e47a2066b842d3d2fe5",
        "RepoTags": [
            "pihole/pihole:latest"
        ],
        "RepoDigests": [
            "pihole/pihole@sha256:8c172c4cf344232a137202811b0c13a0542551076798d7635ef5368f40d1fe71"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2020-02-26T18:43:04.956377056Z",
        "Container": "ea47f3c99ac1ef86d7606b750baf2f9a2c666df8cde3a3e7c9d24148795becfd",
        "ContainerConfig": {
            "Hostname": "ea47f3c99ac1",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "443/tcp": {},
                "53/tcp": {},
                "53/udp": {},
                "67/udp": {},
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "ARCH=arm64",
                "UBUNTU_SUITE=stretch",
                "DOCKER_REPO=multiarch/debian-debootstrap",
                "S6OVERLAY_RELEASE=https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-aarch64.tar.gz",
                "PIHOLE_INSTALL=/root/ph_install.sh",
                "PHP_ENV_CONFIG=/etc/lighttpd/conf-enabled/15-fastcgi-php.conf",
                "PHP_ERROR_LOG=/var/log/lighttpd/error.log",
                "IPv6=True",
                "S6_LOGGING=0",
                "S6_KEEP_ENV=1",
                "S6_BEHAVIOUR_IF_STAGE2_FAILS=2",
                "ServerIP=0.0.0.0",
                "FTL_CMD=no-daemon",
                "DNSMASQ_USER=root",
                "VERSION=v4.4"
            ],
            "Cmd": [
                "/bin/bash",
                "-c",
                "#(nop) ",
                "SHELL [/bin/bash -c]"
            ],
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1"
                ]
            },
            "ArgsEscaped": true,
            "Image": "sha256:16ebd93abfa924e15e01d2fa41458742d4a8b8d0ca2b017db3220f86338182dc",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/s6-init"
            ],
            "OnBuild": [],
            "Labels": {
                "image": "pihole/pihole:v4.4_arm64",
                "maintainer": "adam@diginc.us",
                "url": "https://www.github.com/pi-hole/docker-pi-hole"
            },
            "Shell": [
                "/bin/bash",
                "-c"
            ]
        },
        "DockerVersion": "17.09.0-ce",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "443/tcp": {},
                "53/tcp": {},
                "53/udp": {},
                "67/udp": {},
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/opt/pihole:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "ARCH=arm64",
                "UBUNTU_SUITE=stretch",
                "DOCKER_REPO=multiarch/debian-debootstrap",
                "S6OVERLAY_RELEASE=https://github.com/just-containers/s6-overlay/releases/download/v1.22.1.0/s6-overlay-aarch64.tar.gz",
                "PIHOLE_INSTALL=/root/ph_install.sh",
                "PHP_ENV_CONFIG=/etc/lighttpd/conf-enabled/15-fastcgi-php.conf",
                "PHP_ERROR_LOG=/var/log/lighttpd/error.log",
                "IPv6=True",
                "S6_LOGGING=0",
                "S6_KEEP_ENV=1",
                "S6_BEHAVIOUR_IF_STAGE2_FAILS=2",
                "ServerIP=0.0.0.0",
                "FTL_CMD=no-daemon",
                "DNSMASQ_USER=root",
                "VERSION=v4.4"
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "dig +norecurse +retry=0 @127.0.0.1 pi.hole || exit 1"
                ]
            },
            "ArgsEscaped": true,
            "Image": "sha256:16ebd93abfa924e15e01d2fa41458742d4a8b8d0ca2b017db3220f86338182dc",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/s6-init"
            ],
            "OnBuild": [],
            "Labels": {
                "image": "pihole/pihole:v4.4_arm64",
                "maintainer": "adam@diginc.us",
                "url": "https://www.github.com/pi-hole/docker-pi-hole"
            },
            "Shell": [
                "/bin/bash",
                "-c"
            ]
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 334648690,
        "VirtualSize": 334648690,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/media/storage-hdd/docker/var/overlay2/59cefaa9d6a59a83ba27c8fc84b016e818e623ba3199df324b239b0c7573deb4/diff:/var/media/storage-hdd/docker/var/overlay2/b66c68e331f3327b8a10c78c643616eb857c9267920db48bb61c2108ee12edfe/diff:/var/media/storage-hdd/docker/var/overlay2/de0cd6d122d93c4a8e6c51388250e25a70df31f7c106b85010fb29301a63b232/diff:/var/media/storage-hdd/docker/var/overlay2/e412b0131c79b5de8fcb0313fb3d9710a03e00fe306ab2ffa82dd4f0f27da446/diff:/var/media/storage-hdd/docker/var/overlay2/5f6d5b4adac558debeaf35d88d870f2fad7c9b3396da2186b7fb1341382c1e7a/diff:/var/media/storage-hdd/docker/var/overlay2/e2fe6a9e8684c2280ae58e8868045bc993f6f434ac293fe9ef5123330722adcb/diff:/var/media/storage-hdd/docker/var/overlay2/61bdb18dd96184ded3d839e048a773c1252c2e739e58da284733017f00eded89/diff",
                "MergedDir": "/var/media/storage-hdd/docker/var/overlay2/3f3f8cdda1acb6eeea6090fed2a9015443f7d3e2106b05b0e27bb097eaaa6de1/merged",
                "UpperDir": "/var/media/storage-hdd/docker/var/overlay2/3f3f8cdda1acb6eeea6090fed2a9015443f7d3e2106b05b0e27bb097eaaa6de1/diff",
                "WorkDir": "/var/media/storage-hdd/docker/var/overlay2/3f3f8cdda1acb6eeea6090fed2a9015443f7d3e2106b05b0e27bb097eaaa6de1/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:885e2a16cf220cad321cf56e10b8647e9edae204d6a2dae1e18085a539722b9f",
                "sha256:377b70f809ea80df99fbff026b0ea417b05604b92fa5bf9c2d7ec5299e3fdb5b",
                "sha256:5f75802b6b8c66f7dc5e8838b42ac8d3e6c7a48943c07a2cc7c3ff2d9d5ac5a3",
                "sha256:2829e5b00ce5603fb8fd447f147db495edc64865c8de03c656cc2155bbf4a6ea",
                "sha256:c0e711585b5de43ce9c3e457f72048e0760943d0ba128dbdbe7c9e9f26cdfec7",
                "sha256:4f3435f2ba8a98159ae775f3fb7c71a7f4cac080191203a2162dc69a8faa370b",
                "sha256:7f2223904947849b5e0cacefa945eed8085026fc90e4a1ab0aa939fc252231cd",
                "sha256:0f93307acd3f91fd812702d9dc31ca4230a1be3b944df7f8773e09652d8e6903"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]
Mellbourn commented 4 years ago

@bugsyb Thank you for your patience, but unfortunately it did not work. I ran the commands you suggested, even removed /var/lib/docker, restarted docker daemon, did the pull. Got the same digest as before, the last command gives no output, just error code 159.

> docker stop pihole; docker container rm pihole; docker image rm pihole/pihole
Error response from daemon: No such container: pihole
Error: No such container: pihole
Untagged: pihole/pihole:latest
Untagged: pihole/pihole@sha256:8c172c4cf344232a137202811b0c13a0542551076798d7635ef5368f40d1fe71
Deleted: sha256:505ad79f9e8ad07ae4f5d3c470d77cf9c235f04785854e47a2066b842d3d2fe5
Deleted: sha256:981fa4205bf09bc9b5a8656e1e61ecd55834cfe1ccbd38cb8082056f2a186b80
Deleted: sha256:f73c787533e1cd298deac00a90c7b83456cd8212dff5ad7bb10dd7a95defcc06
Deleted: sha256:2b415f18d239ee0b274c2bf04a241ef2ba597f5c235d0ecaa647ec7d6be0b03c
Deleted: sha256:39e2f8763630c115bd2048369663d041a4d369ab5233f85b861cc361b9791611
Deleted: sha256:10839c8bac12f7a0159a7bbed71dfad19537b7cea1d16fe63e208cfd6dd5575d
Deleted: sha256:2209d5412ffaa4b61615e27dfeff2258e97c328fbff889236a760699a9664bd9
Deleted: sha256:02f5c2dd32e3ab625c4ec0cd2f51e46908c392221f7a119fbfacce8b9d387e9f
Deleted: sha256:885e2a16cf220cad321cf56e10b8647e9edae204d6a2dae1e18085a539722b9f
> sudo systemctl stop docker
> sudo \rm -rf /var/lib/docker
> sudo systemctl start docker
> docker pull --platform="arm64" pihole/pihole:latest
latest: Pulling from pihole/pihole
5ed7afaa9adf: Pull complete
9a1f55b99947: Pull complete
722d4e0ba6ff: Pull complete
164ec22e03b6: Pull complete
f67b9381243f: Pull complete
e44673a7b448: Pull complete
1f47684a5487: Pull complete
f21e2c89ca46: Pull complete
Digest: sha256:8c172c4cf344232a137202811b0c13a0542551076798d7635ef5368f40d1fe71
Status: Downloaded newer image for pihole/pihole:latest
docker.io/pihole/pihole:latest
> docker run --platform="arm64" -it --rm --entrypoint sh pihole/pihole:latest -c "pihole -v; dpkg --print-architecture"
159 >

If I try to docker-compose up -d it goes into a restart loop as usual.

diginc commented 4 years ago

It might be related to a publishing bug, I'm still not 100% sure. I made a small deployment change that should be released to pihole/pihole:dev. hub link for arm64

 $ docker pull pihole/pihole:dev
dev: Pulling from pihole/pihole
6d28e14ab8c8: Already exists
288c0d516484: Pull complete
aa113f1706df: Pull complete
abb9c2faee3c: Pull complete
f84b063eb6ef: Pull complete
b4630b351376: Pull complete
e39542fff870: Pull complete
b6288b96ec91: Pull complete
3fe34f8a8fe9: Pull complete
dd3ab6df8947: Pull complete
7b6dda87ceb8: Pull complete
4ddf832f394d: Pull complete
e0813ded6b2b: Pull complete
Digest: sha256:3a4c8cc914b3e39ed9761bfdd51f8f947e4c75073493aa31d183a1be9f808b59
Status: Downloaded newer image for pihole/pihole:dev
docker.io/pihole/pihole:dev
$ docker manifest inspect pihole/pihole:dev
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 3035,
         "digest": "sha256:fde12c0f51b1e91570469459226a7f0f14602b24e9bcf55b9988e4cfb33f7776",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1987,
         "digest": "sha256:1e72dfb1417e31f60aebb85ead07160f8e8223990ea4a84622e52979936d52d2",
         "platform": {
            "architecture": "arm64",
            "os": "linux",
            "variant": "v8"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1987,
         "digest": "sha256:7f695b73bb747f944e557849a44173dfca00f87069bd8706b02ac12f7cc2381f",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v6"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1987,
         "digest": "sha256:acc29cadc27968c0d378863d032dc89e1c04ebd79a9ef89d51887aa6e99ab342",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      }
   ]
}
Mellbourn commented 4 years ago

@diginc I tested changing image: pihole/pihole:latest to image: pihole/pihole:dev in my docker-compose.yml but it didn't make any difference.

I emptied the cache, removed /var/lib/docker, restarted the daemon, did a docker-compose pull and docker-compose up -d. When the container had started it still had the strange FTL Version:

Pi-hole Version v4.4 Web Interface Version v4.3.3 FTL Version vDev (development, vDev-7dea0f0)

It is still listing Domains on Blocklist as -2.

But I do seem to get the same digest as you:

> docker images --digests
REPOSITORY          TAG                 DIGEST                                                                    IMAGE ID            CREATED             SIZE
pihole/pihole       dev                 sha256:3a4c8cc914b3e39ed9761bfdd51f8f947e4c75073493aa31d183a1be9f808b59   65b1970d504f        4 days ago          324MB
chuegel commented 4 years ago

Have the same issue. I'm using pihole on Odroid running libreelec. 

REPOSITORY          TAG                 DIGEST                                                                    IMAGE ID            CREATED             SIZE
pihole/pihole       latest              sha256:8c172c4cf344232a137202811b0c13a0542551076798d7635ef5368f40d1fe71   623d099087d5        3 weeks ago         324MB
pihole/pihole       v4.4                sha256:8c172c4cf344232a137202811b0c13a0542551076798d7635ef5368f40d1fe71   623d099087d5        3 weeks ago         324MB

What's weird is that a dev version of FTL is being pulled:

docker exec -it pihole pihole version
  Pi-hole version is v4.4 (Latest: v4.4)
  AdminLTE version is v4.3.3 (Latest: v4.3.3)
  FTL version is vDev-7dea0f0 (Latest: v4.3.1)

77369563-3531d380-6d5f-11ea-9de1-ad71bfa16183

bugsyb commented 4 years ago

Guys, I've re-run into this very same issue. Tried both :latest, :v4.4, :release-v4.3.4, removed folders, etc.

Further investigation shown that Gravity listy parsing raised plenty of errors even if sites were correct (check your logs).

The only way to solve it was to use steps from my earlier post, i.e. https://github.com/pi-hole/docker-pi-hole/issues/587#issuecomment-599778560

This is literally down to forcing the architecture/platform and requires to turn on the experimental flag on docker host/containerd side.

Can't prove it, but belief is that one of the binaries included in non arm64 platforms does execute wrong code/compiled for different platform and as such fails. Something to do with Gravity list hence all the problems. With default platform, I've had also all sorts of weird versions reported at the bottom. The moment I've switched to the arm64, all works flawlessly.

julianzeidler commented 4 years ago

Hi I also ran into the same issue on the armel issues (piZero)

I tried all the available armel images (up to dev-armel) but with all of them there is no blocking and it reports an unexpected FTL Version vDev (development, vDev-fb542a7)

I always started with a clean slate and the default recommended docker parameters. I have been running an odler pihole docker on this machine for ~a year so the settings are sound. https://tricorder.pi-hole.net/rbsva30wtk

Also building version tag/v4.4 localy gives the same results

hugokrantz commented 4 years ago

It took me some time before my troubleshooting led me here. Glad to see I'm not alone with this issue, but sad to see that there doesn't seem to be a remedy. :(

diginc commented 4 years ago

I worked around not having any real arm v8 hardware to test on by spinning up an AWS ARM instance and fresh installing everything there...bad news, I couldn't reproduce :( . Here's a dump of steps to reproduce and actually get blocked entries:

AWS Info:

History dump edited to remove any bloat

# Use API since I only opened SSH up on the security group
telnet 127.0.0.1 4711 

>stats
domains_being_blocked 127954
dns_queries_today 210
ads_blocked_today 100
ads_percentage_today 47.619049
...

Image info:  sha256:8c172c4cf344232a137202811b0c13a0542551076798d7635ef5368f40d1fe71   505ad79f9e8a

Could anyone with a faulty container try running this test in the same way I did and report if it fails also?

https://gist.github.com/diginc/ced3f34d4c62df08fe1fb3055ed17382#file-arm64-pihole-history-sh-L35-L39

I tested just my local dig commands, not any remote clients, since I was on AWS with the firewall blocking all traffic but SSH...maybe that's the key difference? Otherwise my mind goes to hardware differences?

I included as much info as possible in the GIST for further debugging collaboration.

PaPaTheGMan commented 4 years ago

Experienced the same problem after Watchtower updated me to 4.4. What worked for me was reverting back 4.3.2-1. I will just leave it there for now.

basrieter commented 4 years ago

Same issue here going from Pi-Hole v4.3.2-1 to latest on Docker running on a Raspberry Pi 3

chuegel commented 4 years ago

reverted back to 4.3.2-1 for now

hugokrantz commented 4 years ago

I also reverted to 4.3.2-1. Works fine.

pralor-bot commented 4 years ago

This issue has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/upgraded-container-no-longer-have-blocked-ads/30463/4

kenperkins commented 4 years ago

@pralor that's from me. I don't know why I didn't think to post here other than I didn't connect this as a potential docker problem.

Exactly as described above; watchtower updated me to 4.4, my FTL has the dev version.

kenperkins commented 4 years ago

Reverting back to 4.3.2-1 worked, although I'd love to help figure out the root problem with 4.4

joelnb commented 4 years ago

Does anyone have a solution for an older Pi? I have a Raspberry Pi Model B Plus Rev 1.2 and I cannot find any image where blocking actually works - most tags do not have a format compatible with this version of the Pi hardware. I found some tags which do - master-armel or dev-armel - but none where the blocking actually works.

thaBo77 commented 4 years ago

I had the same issue running pi-hole on my Raspi (Hypriot Docker Image). It was on version 1.9.0 (Docker 18.04.0-ce, kernel 4.14.34). After upgrading to the latest version 1.12.0 (Docker 19.03.5, kernel 4.19.75) everything's up again. So I'm not sure where the root cause is but maybe this helps some of you guys.

julianzeidler commented 4 years ago

Does anyone have a solution for an older Pi? I have a Raspberry Pi Model B Plus Rev 1.2 and I cannot find any image where blocking actually works - most tags do not have a format compatible with this version of the Pi hardware. I found some tags which do - master-armel or dev-armel - but none where the blocking actually works. @joelnb I got pihole to run on my pi zero by compiling my own image on the pi. I added two modifications to the trunk version:

  1. Modified install.sh line 27 by adding | sed 's#set -e##g' (before the > redirect) to make this script not abort as soon as there is an error, as this was happening while it tried to restart lighttp during install
  2. I changed the FROM to FROM schachr/raspbian-stretch:latest in Dockerfile_armhf (as the debian base would not correctly run a armhf image on raspberry pi zero/1/2) as then build it with "docker built -f Dockerfile_armhf -t localpiholeraspian:4.4 ." It has been running for weeks without a hitch. If you do not want to base it on the old stretch you can build yopur own base folölowing https://www.michaelmiklis.de/creating-a-raspbian-docker-base-image/ (replacing stretch with buster) but this took a looooong time on my pi zero
diginc commented 4 years ago

I bought a pi4 and dug into this. Before switching from armhf to arm64 raspbian kerenel I made sure it ran the normal armhf image fine and it did. After the kernel upgrade & reboot I saw uname -a reporting aarch64 correctly and the old armhf docker install/container still running before I destroyed it to recreate to simulate your upgrade.

longer story short, I reproduced your weirdness but found the issue is with rasbpian docker arm64/aarch64. That said, you can still run armhf on rasbpian aarch64, I managed to do it using the master-armhf tag which would be similar to 'latest', but slightly less controlled/unstable since it is a branch build technically.

More detail on my testing:

My first instinct after changing out the kerrnel architecture was uninstall docker armhf to install the arm64 version since their site said they supported it. Apt-get wouldn't find the docker-ce package after switching /etc/apt/sources.list.d/docker.list from deb [arch=armhf] https://download.docker.com/linux/raspbian buster stable to deb [arch=arm64] https://download.docker.com/linux/raspbian buster stable...

To verify this I snooped in the debian repo data and it appears there are no packages published (note the 0 bytes next to Contents-arm64 vs armhf.). I'm starting to think that docker's support checkbox in their matrix (which linked to fedora instead of raspbian) was a lie.

I dug into that raspbian 64 kernel test thread you posted looking for mentions of docker and found this supporting evidence:

https://www.raspberrypi.org/forums/viewtopic.php?p=1543802#p1543802

If on Raspbian, armhf Docker won't run arm64 containers. I reported on GitHub: https://github.com/docker/distribution/issues/3008
but it's unlikely to get any attention soon. LXC, systemd-nspawn, or chroot have no trouble spawning a 64-bit environment from a 32-bit userland, so the problem is specific to Docker.

The supposed workaround I mentioned above of running Docker inside a 64-bit chroot doesn't work either. Docker reports an invalid argument if you try to run it inside a chroot.

Your best bet is to use this kernel with a native 64-bit userland distro such as Gentoo, Manjaro, or balena.

As I mentioned earlier armhf works fine on this arm64 kernel still if you specifically call out the pi-hole :master-armhf tag it seems. If for some reason you have to run an arm64 image I'd second that poster's comments of either run one of the other arm64 distros docker's official install says they support or just stick to armhf kernel in raspbian.

basrieter commented 4 years ago

I run Hypriot for Docker and have the exact same issue on rPi3.