Adlists/entries disabled after running pihole-updatelists

I've cleaned the database with the commands as mentioned here before running pihole-updatelists.

After running the pihole-updatelists command successfully with no errors, all the adlists get imported, but some entries and adlists are disabled. This started to happen when I began to use multiple [GroupA_adlists] tags within my configuration file.

Please note that:

;PERSISTENT_GROUP=false
;REQUIRE_COMMENT=true

are commented, and as far as I can tell, these setttings are the only ones that could influence the behavior.

I've ran the debug, and this showed up:

Executing using configuration section "FIREBOG_COLLECTION"...

SQL Query: SELECT * FROM `adlist`
SQL Query: SELECT * FROM `adlist_by_group`
Fetching ADLISTS from 'https://v.firebog.net/hosts/lists.php?type=nocross '... done (44 entries)
Processing...
SQL Query: SELECT * FROM `adlist` WHERE `enabled` = 1 AND `comment` LIKE "%Firebog - collection%"
TypeError: str_replace(): Argument #2 ($replace) must be of type array|string, int given in /usr/local/sbin/pihole-updatelists:1349
Stack trace:
#0 /usr/local/sbin/pihole-updatelists(1349): str_replace()
#1 /usr/local/sbin/pihole-updatelists(1322): LoggedPDOStatement->parseQuery()
#2 /usr/local/sbin/pihole-updatelists(1820): LoggedPDOStatement->execute()
#3 {main}
Releasing lock and removing lockfile: /var/lock/pihole-updatelists.lock

Is this something I can fix? I believe it is in the binary, no?

I've added my config file as an attachment, aswell as a verbose log and a debug log debug.log verbose.log pihole-updatelists.conf.txt

Removed first pictures to keep the page short.

That error is debugON specific, it should be fixed in the dev branch now. In your config the OPTIONAL PARAMETERS part should be BEFORE any section declaration otherwise all variables there will only apply to the last section defined - this might be the cause of your issue.

I've changed the config as per your instructions, but the problem persists.

; Pi-hole's Lists Updater by Jack'lul
; https://github.com/jacklul/pihole-updatelists

; =============================================================
; OPTIONAL PARAMETERS (and their default values)
; To change them you have to uncomment them first (remove prefixing ';')
; =============================================================

; Comment string used to know which domains were created by the script
; You can still add your own comments to individual domains as long
; you keep this string intact
COMMENT="Pihole-UpdateLists"

; Assign additional group to all inserted domains
; To assign only the specified group (do not add to the default) make the number negative
; `0` is the default group, you can view ID of the group in Pi-hole's web interface
; by hovering mouse cursor over group name field on the 'Group management' page
;GROUP_ID=0

; Makes sure domains have the specified group assigned on each run
; This does not prevent you from assigning more groups through the web interface
; but can remove domains from the default group if GROUP_ID is negative number
; Do not enable this when you're running multiple different configs
;PERSISTENT_GROUP=false

; Prevent touching domains not created by this script by comparing comment field
; When disabled any user-created entry will be disabled
;REQUIRE_COMMENT=true

; Update gravity after lists are updated? (runs `pihole updateGravity`)
; When disabled invokes lists reload instead
; Set to 'null' to do nothing
UPDATE_GRAVITY=false

; Show more information while the script is running
;VERBOSE=false

; Show debug messages for troubleshooting purposes
;DEBUG=false

; Maximum time in seconds one list download can take before giving up
; You should increase this when downloads fail because of timeout
;DOWNLOAD_TIMEOUT=60

; Ignore download failures when using multiple lists
; This will cause domains from the lists that failed to download to be disabled
IGNORE_DOWNLOAD_FAILURE=true

; Location of gravity.db file in case you need to change it
;GRAVITY_DB="/etc/pihole/gravity.db"

; Process lockfile to prevent multiple instances of the script from running
; You shouldn't change it - unless `/var/lock` is unavailable
;LOCK_FILE="/var/lock/pihole-updatelists.lock"

; Log console output to file
; In most cases you don't have to set this as you can view the log in the system journal
; Put `-` before path to overwrite file instead of appending to it
;LOG_FILE=""

; Branch to pull remote checksum and update from
;GIT_BRANCH="master"

; =============================================================
; ADLIST COLLECTIONS
; Remote list URL containing list of adlists to import
; =============================================================

[firebog_ticked]
ADLISTS_URL="https://v.firebog.net/hosts/lists.php?type=tick"
GROUP_ID=0
COMMENT="Firebog - ticked"

[firebog_nocross]
ADLISTS_URL="https://v.firebog.net/hosts/lists.php?type=nocross"
GROUP_ID=0
COMMENT="Firebog - nocross"

; =============================================================
; OFFLINE ADLIST COLLECTIONS
; Local list URL containing list of adlists to import
; =============================================================

[advertisements_collection]
ADLISTS_URL="/etc/pihole/blocklists/collections/advertisements.list"
GROUP_ID=0
COMMENT="Advertisements - collection"

[adult_collection]
ADLISTS_URL="/etc/pihole/blocklists/collections/adult.list"
GROUP_ID=-1
COMMENT="Adult - collection"

[entertainment_collection]
ADLISTS_URL="/etc/pihole/blocklists/collections/entertainment.list"
GROUP_ID=-2
COMMENT="Entertainment - collection"

[functional_collection]
ADLISTS_URL="/etc/pihole/blocklists/collections/functional.list"
GROUP_ID=-3
COMMENT="Functional - collection"

[games_collection]
ADLISTS_URL="/etc/pihole/blocklists/collections/games.list"
GROUP_ID=-4
COMMENT="Games - collection"

[news_collection]
ADLISTS_URL="/etc/pihole/blocklists/collections/news.list"
GROUP_ID=-5
COMMENT="News - collection"

[random_collection]
ADLISTS_URL="/etc/pihole/blocklists/collections/random.list"
GROUP_ID=-6
COMMENT="Random - collection"

[search_collection]
ADLISTS_URL="/etc/pihole/blocklists/collections/search.list"
GROUP_ID=-7
COMMENT="Search - collection"

[social_collection]
ADLISTS_URL="/etc/pihole/blocklists/collections/social.list"
GROUP_ID=-8
COMMENT="Social - collection"

; =============================================================
; ADLISTS
; =============================================================

[functional_adlists]
WHITELIST_URL="https://raw.githubusercontent.com/mmotti/pihole-regex/master/whitelist.list"
GROUP_ID=-3
COMMENT="Functional - adlist"

; =============================================================
; DOMAIN LISTS
; This is specifically for handcrafted lists only, do not use regular blocklists here!
; =============================================================

[advertisements_domains]
BLACKLIST_URL="/etc/pihole/blocklists/lists/blacklists/advertisements.list"
WHITELIST_URL="/etc/pihole/blocklists/lists/whitelists/advertisements.list"
REGEX_BLACKLIST_URL="/etc/pihole/blocklists/lists/regex_blacklists/advertisements.list"
REGEX_WHITELIST_URL="/etc/pihole/blocklists/lists/regex_whitelists/advertisements.list"
GROUP_ID=0
COMMENT="Advertisements - domains"

[adult_domains]
BLACKLIST_URL="/etc/pihole/blocklists/lists/blacklists/adult.list"
WHITELIST_URL="/etc/pihole/blocklists/lists/whitelists/adult.list"
REGEX_BLACKLIST_URL="/etc/pihole/blocklists/lists/regex_blacklists/adult.list"
REGEX_WHITELIST_URL="/etc/pihole/blocklists/lists/regex_whitelists/adult.list"
GROUP_ID=-1
COMMENT="Adult - domains"

[entertainment_domains]
BLACKLIST_URL="/etc/pihole/blocklists/lists/blacklists/entertainment.list"
WHITELIST_URL="/etc/pihole/blocklists/lists/whitelists/entertainment.list"
REGEX_BLACKLIST_URL="/etc/pihole/blocklists/lists/regex_blacklists/entertainment.list"
REGEX_WHITELIST_URL="/etc/pihole/blocklists/lists/regex_whitelists/entertainment.list"
GROUP_ID=-2
COMMENT="Entertainment - domains"

[functional_domains]
BLACKLIST_URL="/etc/pihole/blocklists/lists/blacklists/functional.list"
WHITELIST_URL="/etc/pihole/blocklists/lists/whitelists/functional.list"
REGEX_BLACKLIST_URL="/etc/pihole/blocklists/lists/regex_blacklists/functional.list"
REGEX_WHITELIST_URL="/etc/pihole/blocklists/lists/regex_whitelists/functional.list"
GROUP_ID=-3
COMMENT="Functional - domains"

[games_domains]
BLACKLIST_URL="/etc/pihole/blocklists/lists/blacklists/games.list"
WHITELIST_URL="/etc/pihole/blocklists/lists/whitelists/games.list"
REGEX_BLACKLIST_URL="/etc/pihole/blocklists/lists/regex_blacklists/games.list"
REGEX_WHITELIST_URL="/etc/pihole/blocklists/lists/regex_whitelists/games.list"
GROUP_ID=-4
COMMENT="Games - domains"

[news_domains]
BLACKLIST_URL="/etc/pihole/blocklists/lists/blacklists/news.list"
WHITELIST_URL="/etc/pihole/blocklists/lists/whitelists/news.list"
REGEX_BLACKLIST_URL="/etc/pihole/blocklists/lists/regex_blacklists/news.list"
REGEX_WHITELIST_URL="/etc/pihole/blocklists/lists/regex_whitelists/news.list"
GROUP_ID=-5
COMMENT="News - domains"

[random_domains]
BLACKLIST_URL="/etc/pihole/blocklists/lists/blacklists/random.list"
WHITELIST_URL="/etc/pihole/blocklists/lists/whitelists/random.list"
REGEX_BLACKLIST_URL="/etc/pihole/blocklists/lists/regex_blacklists/random.list"
REGEX_WHITELIST_URL="/etc/pihole/blocklists/lists/regex_whitelists/random.list"
GROUP_ID=-6
COMMENT="Random - domains"

[search_domains]
BLACKLIST_URL="/etc/pihole/blocklists/lists/blacklists/search.list"
WHITELIST_URL="/etc/pihole/blocklists/lists/whitelists/search.list"
REGEX_BLACKLIST_URL="/etc/pihole/blocklists/lists/regex_blacklists/search.list"
REGEX_WHITELIST_URL="/etc/pihole/blocklists/lists/regex_whitelists/search.list"
GROUP_ID=-7
COMMENT="Search - domains"

[social_domains]
BLACKLIST_URL="/etc/pihole/blocklists/lists/blacklists/social.list"
WHITELIST_URL="/etc/pihole/blocklists/lists/whitelists/social.list"
REGEX_BLACKLIST_URL="/etc/pihole/blocklists/lists/regex_blacklists/social.list"
REGEX_WHITELIST_URL="/etc/pihole/blocklists/lists/regex_whitelists/social.list"
GROUP_ID=-8
COMMENT="Social - domains"

Here's my updated config, and again some pictures: pihole-updatelists.conf.txt domains adlists

As far as I can tell from the verbose log nothing got disabled there, could you repost newer debug log (enable some disabled entries before)?

You have really a lot of sections and I guess one/some of them is breaking your setup...

I checked all the sections, I personally could not find any faults with the syntax. I will remake the file somewhere this weekend to see if it helps though.

I have also used timeshift to restore the system to a point that there was nothing installed that has to do with pihole, and I reinstalled pihole with pihole-updatelists. Still the same situation regarding disabled lists and domains.

I've added the logs. pihole-updatelists-debug.log pihole-updatelists-verbose.log

I think one of the issues might be having multiple section assigned to same group ID, from my experience firebog/ticked and firebog/nocross will always conflict with each other because they contain common entries. Even if both sections have different comment string the script will still recognize entries of the other sections as "touchable" thus disabling them when needed. You might try splitting the config into multiple ones.

You are right, after commenting out:

[firebog_ticked]
ADLISTS_URL="https://v.firebog.net/hosts/lists.php?type=tick"
GROUP_ID=0
COMMENT="Firebog - ticked"

and:

; =============================================================
; ADLISTS
; =============================================================

[functional_adlists]
WHITELIST_URL="https://raw.githubusercontent.com/mmotti/pihole-regex/master/whitelist.list"
GROUP_ID=-3
COMMENT="Functional - adlist"

Everything showed up as enabled again. Is this behavior intended?

Everything showed up as enabled again. Is this behavior intended?

Yes, there is mention in the readme that this is not supported. Maybe it could work if you put different GROUP_ID for those conflicting entries.

I've managed to make it work with multiple config files.

I could not find anything about duplicate group entries for these variables in the readme:

ADLISTS_URL=
WHITELIST_URL=
REGEX_WHITELIST_URL=
BLACKLIST_URL=
REGEX_BLACKLIST_URL=

It seems that adding a duplicate of one of the variables mentioned above to the same config file with the same GROUP_ID causes conflict. This causes conflict, for example:

[GroupA_adlists]
ADLISTS_URL='http://adlist1.url"
GROUP_ID=1
COMMENT="pihole-updatelists - Whitelist A"

[GroupB_adlists]
ADLISTS_URL='http://adlist2.url"
GROUP_ID=1
COMMENT="pihole-updatelists - Whitelist B"

But for example, this causes no conflict, since the variables aren't the same:

[GroupA_adlists]
ADLISTS_URL='http://adlist1.url"
GROUP_ID=1
COMMENT="pihole-updatelists - Adlists A"

[GroupA_whitelist]
WHITELIST_URL='http://whitelist1.url"
GROUP_ID=1
COMMENT="pihole-updatelists - Whitelist A"

# Different Group ID also works
[GroupB_adlist]
ADLISTS_URL='http://adlist2.url"
GROUP_ID=2
COMMENT="pihole-updatelists - Adlists B"

In the readme it does mention: Multiple groups are not supported But I figure this means:

[GroupA_whitelist]
WHITELIST_URL='http://url.url"
GROUP_ID=1,2,3
GROUP_ID=4
GROUP_ID=5
COMMENT="pihole-updatelists - Whitelist A"

Perhaps the group conflict issue should be documented into the readme?

Anyways, enough about the documentation. I have set up multiple configuration files, and I wish to execute them with the pihole-updatelists timer.

I am confused about the schedule and the scheduled custom commands settings.

when I run: sudo systemctl edit pihole-updatelists.timer the file shows:

### Editing /etc/systemd/system/pihole-updatelists.timer.d/override.conf
### Anything between here and the comment below will become the new contents of the file

### Lines below this comment will be discarded

### /etc/systemd/system/pihole-updatelists.timer
# [Unit]
# Description=Weekly update of Pi-hole's lists
# 
# [Timer]
# RandomizedDelaySec=60m
# OnCalendar=Sat *-*-* 03:00:00
# Persistent=true
# 
# [Install]
# WantedBy=timers.target

Yet in your readme section it says to add:

[Timer]
RandomizedDelaySec=5m
OnCalendar=
OnCalendar=Sat *-*-* 00:00:00

What is the reason for the duplicate OnCalendar variable, with the first one being empty? I assume this is a typo? And why is the Persistent tag missing? Does the override timer unit settings file inherit this variable from the original timer unit file?

Anyways, would this be correct? Timer unit config:

### Editing /etc/systemd/system/pihole-updatelists.timer.d/override.conf
### Anything between here and the comment below will become the new contents of the file

[Timer]
RandomizedDelaySec=5m
OnCalendar=Sat *-*-* 00:00:00
Persistent=true

### Lines below this comment will be discarded

### /etc/systemd/system/pihole-updatelists.timer
# [Unit]
# Description=Weekly update of Pi-hole's lists
# 
# [Timer]
# RandomizedDelaySec=60m
# OnCalendar=Sat *-*-* 03:00:00
# Persistent=true
# 
# [Install]
# WantedBy=timers.target

Service unit config:

### Editing /etc/systemd/system/pihole-updatelists.service.d/override.conf
### Anything between here and the comment below will become the new contents of the file

[Service]
Type=oneshot
ExecStartPost=/path/to/shellscript/with/pihole-updatelists/commands/with/custom/config

### Lines below this comment will be discarded

### /etc/systemd/system/pihole-updatelists.service
# [Unit]
# Description=Update Pi-hole's lists from remote sources
# After=network-online.target multi-user.target
# Wants=network-online.target
# 
# [Service]
# Type=simple
# ExecStart=/usr/local/sbin/pihole-updatelists
# TimeoutStartSec=infinity

This is where a "kinda" mention about it is:

What is the reason for the duplicate OnCalendar variable, with the first one being empty? I presume this is a typo?

If you don't start with empty one you're adding another schedule, empty one prevents the schedule from the base file from being used.

And why is the Persistent tag missing? Does the override timer unit settings file inherit this variable from the original timer unit file?

That command creates an override file which is merged with the base file, you don't have to re-add thing that are in base file already. You can learn more about systemd overrides by searching the web.

Anyways, would this be correct?

Looks fine, though you don't need that Persistent directive.

Service unit config:

What is it supposed to be doing? From what you done it will run /path/to/shellscript/with/pihole-updatelists/commands/with/custom/config command after scheduled run runs successfully.

This is where a "kinda" mention about it is:

Okay, it mentions duplicate comments, but it does not mention anything about duplicate variables with the same GROUP_ID. Of course it is up to you whether you want to add my suggestion to the readme file, or leave it as is. I just wanted to mention it.

What is the reason for the duplicate OnCalendar variable, with the first one being empty? I presume this is a typo?

If you don't start with empty one you're adding another schedule, empty one prevents the schedule from the base file from being used.

Okay, check.

And why is the Persistent tag missing? Does the override timer unit settings file inherit this variable from the original timer unit file?

That command creates an override file which is merged with the base file, you don't have to re-add thing that are in base file already. You can learn more about systemd overrides by searching the web.

Thanks, and you're right, I should have done my research more thoroughly.

Service unit config:

What is it supposed to be doing? From what you done it will run /path/to/shellscript/with/pihole-updatelists/commands/with/custom/config command after scheduled run runs successfully. This is exactly what I want it to do. I now know enough to complete my setup.

Thanks again for your help and information :)

jacklul / pihole-updatelists

Adlists/entries disabled after running pihole-updatelists #118