adlist isn't applying lists

kbreit commented 3 years ago

I have the following information in my values file:

adlists:
  - https://raw.githubusercontent.com/jmdugan/blocklists/master/corporations/facebook/all

It creates the adlist ConfigMap and the /etc/pihole/adlist.list file but I don't see the adlist showing up in the web UI.

MoJo2600 commented 3 years ago

I did a fresh install of pihole today and for me the lists are shown at Group Management -> Adlists

kbreit commented 3 years ago

The ones that do show up are migrated from Pihole 2. Is Pihole able to have migrated and new lists or do I need to start fresh with that?

MoJo2600 commented 3 years ago

You are right, I have to check this. I think pihole will only migrate adlists to the internal database once during an initial startup.

MoJo2600 commented 3 years ago

I checked the documentation and it is like this: Currently we are mounting the adlist file during container startup. Then pihole will attempt to import the adlists to it's database. This is only done once during initial start. If you already have the lists in your database and restart the container, pihole will not add the new lists. You would have to add them in the web interface.

Currently I'm not sure how we could detect the changes and then import them on the fly. Maybe the "right" way to manage the lists in the configmap would be to disable persistence on the container. In this case the database is deleted during shutdown and every restart will trigger a fresh import. But then we would lose all other information as well, like statistics, ect.

Not sure what we could do here.

kbreit commented 3 years ago

@MoJo2600 Do you happen to know how to remove the lists in the database? I can do that then add them in using the adlists.

kbreit commented 3 years ago

I'm trying to delete the database entries from the web UI and receiving an error:

Error, something went wrong! While executing adlist_by_group statement: attempt to write a readonly database

Would you expect this based on the helm chart? Doesn't seem right to me. Troubleshooting this further I am seeing

root@pihole-86d7ddf55c-6gsqz:/etc/pihole# chown pihole.pihole adlists.list
chown: changing ownership of 'adlists.list': Read-only file system

MoJo2600 commented 3 years ago

I tried it on my installation and I was able to delete lists and new lists from Group Management -> Adlists. My storage backend is glusterfs. What is your storage backend? NFS? There is a known issue with NFS. But also a solution here

MoJo2600 commented 3 years ago

@kbreit Is it working for you now? I'd want to close the issue if so.

moritzj29 commented 2 years ago

I encoutered the same issue today. Apparently, as of piHole v5 managing adlists (and other lists) per file is no longer supported: https://discourse.pi-hole.net/t/how-to-update-adlists-from-adlists-list-file/38370

Since the recommended way of managing appears to be the webUI, I would vote for removing the corresponding options in values.yaml. Or at least state somewhere in the documentation that this is deprecated...

Nevertheless, thanks for the helm template, it's been a great help for someone just getting started with k8s!

MoJo2600 commented 2 years ago

Thank you @moritzj29 this is a valid point. I'm in the process in releasing version 3 of this chart with some breaking changes. I think I will remove this setting then.

safehome-jdev commented 6 months ago

@MoJo2600 Is this still a valid, ongoing issue? I found a workaround to be to create an init container, mount the necessary minimums:

configMaps:

setupVars.conf
adlists.list volumeMounts:
/etc/pihole

and run the init-container args of /bin/sh -c "pihole -g -r recreate". This instantiates the gravity.db & pihole-FTL.db databases and necessary files. It then gets handed off the the PiHole container where you would then target those specific files (you could target the whole path of /etc/pihole, but that was out of scope for me).

An example to accomplish this could be:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pihole-test
  annotations:
  labels:
    app: pihole-test
    release: pihole-test
  namespace: pihole
spec:
  selector:
    matchLabels:
      app: pihole-test
      release: pihole-test
  template:
    metadata:
      labels:
        app: pihole-test
        release: pihole-test
      namespace: pihole
    spec:
      containers:
        - name: pihole
          image: pihole/pihole:latest
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 10
            httpGet:
              path: /admin/index.php
              port: http
              scheme: HTTP
            initialDelaySeconds: 60
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          ports:
            - containerPort: 80
              name: http
              protocol: TCP
            - containerPort: 53
              name: dns
              protocol: TCP
            - containerPort: 53
              name: dns-udp
              protocol: UDP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /admin/index.php
              port: http
              scheme: HTTP
            initialDelaySeconds: 60
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          volumeMounts:
            - mountPath: /etc/pihole/adlists.list
              name: adlists
              subPath: adlists.list
            - name: config
              mountPath: /etc/pihole/gravity.db
              subPath: gravity.db
            - name: config
              mountPath: /etc/pihole/pihole-FTL.db
              subPath: pihole-FTL.db
      initContainers:
        - name: init
          imagePullPolicy: IfNotPresent
          active: true
          volumeMounts:
            - name: config
              mountPath: /etc/pihole
            - name: adlists
              mountPath: /etc/pihole/adlists.list
              subPath: adlists.list
            - name: setupvars
              mountPath: /etc/pihole/setupVars.conf
              subPath: setupVars.conf
          image: pihole/pihole
          command:
            - /bin/sh
          tty: false
          args:
            - '-c'
            - pihole -g -r recreate
      volumes:
        - emptyDir:
            medium: ''
          name: config
        - configMap:
            defaultMode: 420
            name: pihole-adlists
          name: adlists
        - configMap:
            defaultMode: 420
            name: pihole-setupvars-test
          name: setupvars
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: pihole-adlists
  namespace: pihole
data:
  adlists.list: |
    https://adaway.org/hosts.txt
    https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt
    https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: pihole-setupvars-test
  namespace: pihole
data:
  setupVars.conf: |
    BLOCKING_ENABLED=true
    CACHE_SIZE=10000
    DNS_BOGUS_PRIV=true
    DNS_FQDN_REQUIRED=true
    DNSMASQ_LISTENING=local
    INSTALL_WEB_INTERFACE=true
    INSTALL_WEB_SERVER=true
    IPV4_ADDRESS=0.0.0.0
    IPV6_ADDRESS=0:0:0:0:0:0
    LIGHTTPD_ENABLED=true
    PIHOLE_DNS_1=1.1.1.1
    PIHOLE_DNS_2=1.0.0.1
    PIHOLE_INTERFACE=all
    QUERY_LOGGING=true

Log output from the init container:

[...]
rm: cannot remove '/etc/pihole/gravity.db': No such file or directory
cp: cannot stat 'migration_backup/*': No such file or directory
  [✓] Recreating gravity database from migration backup
  [i] Creating new gravity database
  [i] Migrating content of /etc/pihole/adlists.list into new database
  [✗] Unable to backup /etc/pihole/adlists.list to /etc/pihole/migration_backup
  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [✓] Creating new gravity databases
  [i] Using libz compression

  [i] Target: https://adaway.org/hosts.txt
  [✓] Status: Retrieval successful
  [✓] Parsed 6540 exact domains and 0 ABP-style domains (ignored 0 non-domain entries)

  [i] Target: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt
  [✓] Status: Retrieval successful
  [✓] Parsed 16712 exact domains and 0 ABP-style domains (ignored 0 non-domain entries)

  [i] Target: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
  [✓] Status: Retrieval successful
  [✓] Parsed 244 exact domains and 0 ABP-style domains (ignored 0 non-domain entries)

  [✓] Building tree
  [✓] Swapping databases
  [✓] The old database remains available
  [i] Number of gravity domains: 23496 (23027 unique domains)
  [i] Number of exact blacklisted domains: 0
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 0
  [i] Number of regex whitelist filters: 0
  [✓] Cleaning up stray matter

  [✗] DNS service is NOT running [<--- we expect this to happen and this is okay]

Log output from Pi-Hole container with only gravity.db & pihole-FTL.db being mounted:

[...]
  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [✓] Creating new gravity databases
  [i] Using libz compression

  [i] Target: https://adaway.org/hosts.txt
  [✓] Status: Retrieval successful
  [✓] Parsed 6540 exact domains and 0 ABP-style domains (ignored 0 non-domain entries)

  [i] Target: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt
  [✓] Status: Retrieval successful
  [✓] Parsed 16712 exact domains and 0 ABP-style domains (ignored 0 non-domain entries)

  [i] Target: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
  [✓] Status: Retrieval successful
  [✓] Parsed 244 exact domains and 0 ABP-style domains (ignored 0 non-domain entries)

  [✓] Building tree
mv: cannot move '/etc/pihole/gravity.db' to '/etc/pihole/gravity_old.db': Device or resource busy
mv: cannot move '/etc/pihole/gravity.db_temp' to '/etc/pihole/gravity.db': Device or resource busy
  [✓] Swapping databases
  [✓] The old database remains available
  [i] Number of gravity domains: 23496 (23027 unique domains)
  [i] Number of exact blacklisted domains: 0
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 0
  [i] Number of regex whitelist filters: 0
  [✓] Cleaning up stray matter

  [✓] FTL is listening on port 53
     [✓] UDP (IPv4)
     [✓] TCP (IPv4)
     [✓] UDP (IPv6)
     [✓] TCP (IPv6)

  [i] Pi-hole blocking will be enabled
  [i] Enabling blocking
  [✓] Pi-hole Enabled

  Pi-hole version is v5.18.2 (Latest: v5.18.2)
  web version is v5.21 (Latest: v5.21)
  FTL version is v5.25.1 (Latest: v5.25.1)
  Container tag is: 2024.03.2

Log output from Pi-Hole container with /etc/pihole being mounted:

[...]
  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [✓] Creating new gravity databases
  [i] Using libz compression

  [i] Target: https://adaway.org/hosts.txt
  [✓] Status: No changes detected
  [✓] Parsed 6540 exact domains and 0 ABP-style domains (ignored 0 non-domain entries)

  [i] Target: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt
  [✓] Status: Retrieval successful
  [✓] Parsed 16712 exact domains and 0 ABP-style domains (ignored 0 non-domain entries)
  [i] List stayed unchanged

  [i] Target: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
  [✓] Status: Retrieval successful
  [✓] Parsed 244 exact domains and 0 ABP-style domains (ignored 0 non-domain entries)
  [i] List stayed unchanged

  [✓] Building tree
  [✓] Swapping databases
  [✓] The old database remains available
  [i] Number of gravity domains: 23496 (23027 unique domains)
  [i] Number of exact blacklisted domains: 0
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 0
  [i] Number of regex whitelist filters: 0
  [✓] Cleaning up stray matter

  [✓] FTL is listening on port 53
     [✓] UDP (IPv4)
     [✓] TCP (IPv4)
     [✓] UDP (IPv6)
     [✓] TCP (IPv6)

  [i] Pi-hole blocking will be enabled
  [i] Enabling blocking
  [✓] Pi-hole Enabled

  Pi-hole version is v5.18.2 (Latest: null)
  FTL version is v5.25.1 (Latest: null)
  Container tag is: 2024.03.2

In my experience, I don't care that there's no backup of the databases since they'll get recreated on crash/redeployment (mv: cannot move '/etc/pihole/gravity.db' to '/etc/pihole/gravity_old.db': Device or resource busy) . But that may bother some. Let me know if I can provide more context/help!

MoJo2600 commented 6 months ago

Init container sounds like a valid approach. I thought they are a beta feature or something? I'm not 100% familiar with the kubernetes feature timeline atm, sorry.

safehome-jdev commented 6 months ago

@MoJo2600 As far as I am aware, initContainers have been around for quite some time if not since it was released initially (someone can correct me though), so unfortunately I can't answer that with 100% confidence. Having said that, it looks like one might be able to inject an initContainer right into your current chart using the initContainer field. You just need to state the right parameters to make sure one gets it right. You need:

extraVolumes:

config: used to temporarily persist data to pod.

Note: In my experience, using the emptyDir's Memory medium for storage resulted in a read-only path error and prevented Pi-Hole from generating the necessary gravity.db & pihole-FTL.db. You must use the node's default medium by leaving this as an empty object or not expressing the extraVolumes.emptyDir.medium field)

extraVolumeMounts:

config: used to mount initContainer changes to the pihole container

configMap:

setupvars: used to provide required parameters to Pi-Hole on startup

An example could be:

[...]
extraInitContainers:
  - active: true
    args:
      - '-c'
      - pihole -g -r recreate
    command:
      - /bin/sh
    image: pihole/pihole
    imagePullPolicy: IfNotPresent
    name: init
    tty: false
    volumeMounts:
      - mountPath: /etc/pihole
        name: config
      - mountPath: /etc/pihole/adlists.list
        name: adlists
        subPath: adlists.list
      - mountPath: /etc/pihole/setupVars.conf
        name: setupvars
        subPath: setupVars.conf
extraObjects:
  - apiVersion: v1
    kind: ConfigMap
    metadata:
      name: pihole-setupvars
      namespace: pihole
    data:
      setupVars.conf: |
        BLOCKING_ENABLED=true
        CACHE_SIZE=10000
        DNS_BOGUS_PRIV=true
        DNS_FQDN_REQUIRED=true
        DNSMASQ_LISTENING=local
        INSTALL_WEB_INTERFACE=true
        INSTALL_WEB_SERVER=true
        IPV4_ADDRESS=0.0.0.0
        IPV6_ADDRESS=0:0:0:0:0:0
        LIGHTTPD_ENABLED=true
        PIHOLE_DNS_1=1.1.1.1
        PIHOLE_DNS_2=1.0.0.1
        PIHOLE_INTERFACE=all
        QUERY_LOGGING=true
extraVolumeMounts:
  config:
    mountPath: /etc/pihole
extraVolumes:
  config:
    emptyDir: {}
  adlists:
    configMap:
      name: pihole-adlists
  setupvars:
    configMap:
      name: pihole-setupvars
[...]

edit: added info/purpose for extraVolumeMounts

MoJo2600 / pihole-kubernetes

adlist isn't applying lists #134