Closed shamoon closed 7 months ago
Pinging @PromoFaux as this is an issue happening in docker
, but we can keep this PR here as a fix for FTL will be needed to avoid this in the future.
Unfortunately, it isn't easy to tell what is happening here, a first suggestions seems to be here:
2024-03-28 11:56:22 2024-03-28 11:56:21.542 [150/T152] WARNING: Disk shortage ahead: 100% is used (18.4EB used, 494.4GB total) on fakeowner filesystem mounted at /etc/pihole
with emphasis on the
18.4EB used, 494.4GB total
part. I recall we have seen this on other MacOS systems before... Maybe this "fakeowner
" filesystem (have never seen this before) is causing issues.
Could you test if the container is still crashing when you don't mount the /etc/pihole
directory externally. I know that this is not a permanent solution but it'd tremendously simplify debugging when we can isolate the cause.
Thanks for your reply! You're correct, that works just fine (I could've sworn I tried that, but I guess not 😅). If its relevant, here's the full log: https://gist.github.com/shamoon/e438f0c692fcebef9bac47d36ff51c82
Please let me know if theres anything else I can help to debug
More clues here, perhaps : https://github.com/pi-hole/docker-pi-hole/issues/951
@shamoon could you please again test the crashing case, but this time with the environment variable
FTLCONF_debug_gc=true
being set. Should give us a bit more output right before the crash happens...
I also have a potential fix available on FTL branch fix/check_disk
, I'm not quite sure how to get this into the v6 docker container for you. I know, I have asked this quite often but is there any more user-friendly version than cloning the repo and re-building locally @PromoFaux ?
All the instructions for building an image with alternative branches is in the Readme of the development-v6 branch of the docker repo :)
Thanks guys, I can confirm that fix works, amazing! Logs are below in case theyre useful.
and
Thanks for your confirmation. The fix is restoring what we had before (and accidentally removed during some code refactoring): Ignoring what macOS reports (exabytes used on a system that is only gigabytes in size).
If you are willing to do so, we can make another attempt at trying to fix this macOS issues. If you could grab the latest binary from this branch, you should see log lines like
DEBUG_GC: Statvfs() results for /etc/pihole/pihole-FTL.db:
DEBUG_GC: (some interesting details here)
right after the line
DEBUG_GC: Checking free space at /etc/pihole/pihole-FTL.db: 100% > 90%
It'd be great if you could then run and post the output of
stat /etc/pihole/
inside the container and - if macOS allows to - also on the host system for the directory containing the mounted data.
I'm looking for some kind of "fixable" weirdness, e.g. a byte-order issue or some incorrectly set bits that we could detect and undo to regain the real numbers. So far, we know this is happening on macOS as host both with fakeowner
and also grpcfuse
(seen in https://github.com/pi-hole/docker-pi-hole/issues/951)
Of course!
/ # stat /etc/pihole/
File: /etc/pihole/
Size: 704 Blocks: 0 IO Block: 4096 directory
Device: 0,68 Inode: 8 Links: 22
Access: (0755/drwxr-xr-x) Uid: ( 100/ pihole) Gid: ( 101/ pihole)
Access: 2024-03-31 20:26:37.839184161 -0700
Modify: 2024-03-31 20:26:37.656822002 -0700
Change: 2024-03-31 20:26:37.656822002 -0700
Birth: -
And on my machine:
stat '.../piholev6/etc-pihole'
16777233 227998945 drwxr-xr-x 22 myusername staff 0 704 "Mar 31 20:26:37 2024" "Mar 31 20:26:37 2024" "Mar 31 20:26:37 2024" "Mar 28 11:46:17 2024" 4096 0 0 .../piholev6/etc-pihole
Now we see what the issue is: statvfs
is reporting a very large block size - but only inside the container:
Block size: 1048576
whereas outside of the container, the blocksize is reported to be 4096
(the typical default since the days of 32bit systems).
1048576
is 0x4000000
- this is 1 MiB which is pretty huge as block size. The solution will be using only the fragment size which is typically the same on *nix systems, but apparently not always - and not on macOS (= BSD similar), see also https://github.com/torvalds/linux/blame/39cd87c4eb2b893354f3b850f916353f2658ae6f/fs/nfs/super.c#L278-L283 (last change to this comment was > 18 years ago)
Would be interesting to see if the latest commit I just pushed to this branch fixes the humongous/incorrect filesystem size on the directory mounted from macOS into your container.
Amazing torvalds link
Seems like a crash now, here's the full output:
2024-04-01 00:19:29 [i] Starting docker specific checks & setup for docker pihole/pihole
2024-04-01 00:19:30
2024-04-01 00:19:30 [i] Ensuring basic configuration by re-running select functions from basic-install.sh
2024-04-01 00:19:30 [i] Assigning password defined by Environment Variable
2024-04-01 00:19:32 [i] Docker start setup complete
2024-04-01 00:19:32
2024-04-01 00:19:32 [i] pihole-FTL (no-daemon) will be started as pihole
2024-04-01 00:19:32
2024-04-01 00:19:32 [i] Setting capabilities on pihole-FTL where possible
2024-04-01 00:19:32 [i] Applying the following caps to pihole-FTL:
2024-04-01 00:19:32 * CAP_CHOWN
2024-04-01 00:19:32 * CAP_NET_BIND_SERVICE
2024-04-01 00:19:32 * CAP_NET_RAW
2024-04-01 00:19:32
2024-04-01 00:19:34 2024-04-01 00:19:33.027 [143M] INFO: ########## FTL started on 3f6c28fb4f2e! ##########
2024-04-01 00:19:34 2024-04-01 00:19:33.027 [143M] INFO: FTL branch: fix/check_disk
2024-04-01 00:19:34 2024-04-01 00:19:33.027 [143M] INFO: FTL version: vDev-90dda14
2024-04-01 00:19:34 2024-04-01 00:19:33.027 [143M] INFO: FTL commit: 90dda142
2024-04-01 00:19:34 2024-04-01 00:19:33.027 [143M] INFO: FTL date: 2024-04-01 09:11:06 +0200
2024-04-01 00:19:34 2024-04-01 00:19:33.027 [143M] INFO: FTL user: pihole
2024-04-01 00:19:34 2024-04-01 00:19:33.027 [143M] INFO: Compiled for linux/arm64/v8 (compiled on CI) using cc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: ************************
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * DEBUG SETTINGS *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * DATABASE: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * NETWORKING: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * LOCKS: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * QUERIES: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * FLAGS: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * SHMEM: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * GC: YES *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * ARP: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * REGEX: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * API: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * TLS: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * OVERTIME: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * STATUS: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * CAPS: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * DNSSEC: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * VECTORS: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * RESOLVER: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * EDNS0: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * CLIENTS: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * ALIASCLIENTS: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * EVENTS: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * HELPER: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * CONFIG: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * INOTIFY: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * WEBSERVER: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * EXTRA: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: * RESERVED: NO *
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] DEBUG_ANY: ************************
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] INFO: 4 FTLCONF environment variables found (4 used, 0 invalid, 0 ignored)
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] INFO: [✓] FTLCONF_dns_listeningMode is used
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] INFO: [✓] FTLCONF_webserver_api_password is used
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] INFO: [✓] FTLCONF_debug_gc is used
2024-04-01 00:19:34 2024-04-01 00:19:33.261 [143M] INFO: [✓] FTLCONF_dns_upstreams is used
2024-04-01 00:19:34 2024-04-01 00:19:33.267 [143M] INFO: Wrote config file:
2024-04-01 00:19:34 2024-04-01 00:19:33.267 [143M] INFO: - 135 total entries
2024-04-01 00:19:34 2024-04-01 00:19:33.267 [143M] INFO: - 127 entries are default
2024-04-01 00:19:34 2024-04-01 00:19:33.267 [143M] INFO: - 8 entries are modified
2024-04-01 00:19:34 2024-04-01 00:19:33.267 [143M] INFO: - 3 entries are forced through environment
2024-04-01 00:19:34 2024-04-01 00:19:33.274 [143M] INFO: Config file written to /etc/pihole/pihole.toml
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] INFO: Parsed config file /etc/pihole/pihole.toml successfully
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] WARNING: Cannot set process priority to -10: Permission denied. Process priority remains at 0
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Statvfs() results for /dev/shm:
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Block size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Fragment size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Total blocks: 16384
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Free blocks: 16384
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Available blocks: 16384
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Total inodes: 1004892
2024-04-01 00:19:33 chmod: cannot read directory '/etc/pihole': Permission denied
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Free inodes: 1004891
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Available inodes: 1004891
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Filesystem ID: 135079855
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Mount flags: 4142
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Maximum filename length: 255
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Statvfs() results for /dev/shm:
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Block size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Fragment size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Total blocks: 16384
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Free blocks: 16383
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Available blocks: 16383
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Total inodes: 1004892
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Free inodes: 1004890
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Available inodes: 1004890
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Filesystem ID: 135079855
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Mount flags: 4142
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Maximum filename length: 255
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Statvfs() results for /dev/shm:
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Block size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Fragment size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Total blocks: 16384
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Free blocks: 16382
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Available blocks: 16382
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Total inodes: 1004892
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Free inodes: 1004889
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Available inodes: 1004889
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Filesystem ID: 135079855
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Mount flags: 4142
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Maximum filename length: 255
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Statvfs() results for /dev/shm:
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Block size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Fragment size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Total blocks: 16384
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Free blocks: 16381
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Available blocks: 16381
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Total inodes: 1004892
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Free inodes: 1004888
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Available inodes: 1004888
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Filesystem ID: 135079855
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Mount flags: 4142
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Maximum filename length: 255
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Statvfs() results for /dev/shm:
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Block size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Fragment size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Total blocks: 16384
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Free blocks: 16371
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Available blocks: 16371
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Total inodes: 1004892
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Free inodes: 1004887
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Available inodes: 1004887
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Filesystem ID: 135079855
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Mount flags: 4142
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Maximum filename length: 255
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Statvfs() results for /dev/shm:
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Block size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Fragment size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Total blocks: 16384
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Free blocks: 16370
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Available blocks: 16370
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Total inodes: 1004892
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Free inodes: 1004886
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Available inodes: 1004886
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Filesystem ID: 135079855
2024-04-01 00:19:34 2024-04-01 00:19:33.277 [143M] DEBUG_GC: Mount flags: 4142
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Maximum filename length: 255
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Statvfs() results for /dev/shm:
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Block size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Fragment size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Total blocks: 16384
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Free blocks: 16349
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Available blocks: 16349
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Total inodes: 1004892
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Free inodes: 1004885
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Available inodes: 1004885
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Filesystem ID: 135079855
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Mount flags: 4142
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Maximum filename length: 255
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Statvfs() results for /dev/shm:
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Block size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Fragment size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Total blocks: 16384
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Free blocks: 16342
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Available blocks: 16342
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Total inodes: 1004892
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Free inodes: 1004884
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Available inodes: 1004884
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Filesystem ID: 135079855
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Mount flags: 4142
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Maximum filename length: 255
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Statvfs() results for /dev/shm:
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Block size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Fragment size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Total blocks: 16384
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Free blocks: 16270
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Available blocks: 16270
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Total inodes: 1004892
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Free inodes: 1004883
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Available inodes: 1004883
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Filesystem ID: 135079855
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Mount flags: 4142
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Maximum filename length: 255
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Statvfs() results for /dev/shm:
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Block size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Fragment size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Total blocks: 16384
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Free blocks: 16268
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Available blocks: 16268
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Total inodes: 1004892
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Free inodes: 1004882
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Available inodes: 1004882
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Filesystem ID: 135079855
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Mount flags: 4142
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Maximum filename length: 255
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Statvfs() results for /dev/shm:
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Block size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Fragment size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Total blocks: 16384
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Free blocks: 16265
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Available blocks: 16265
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Total inodes: 1004892
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Free inodes: 1004881
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Available inodes: 1004881
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Filesystem ID: 135079855
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Mount flags: 4142
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Maximum filename length: 255
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Statvfs() results for /dev/shm:
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Block size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Fragment size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Total blocks: 16384
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Free blocks: 16264
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Available blocks: 16264
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Total inodes: 1004892
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Free inodes: 1004880
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Available inodes: 1004880
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Filesystem ID: 135079855
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Mount flags: 4142
2024-04-01 00:19:34 2024-04-01 00:19:33.278 [143M] DEBUG_GC: Maximum filename length: 255
2024-04-01 00:19:34 2024-04-01 00:19:33.279 [143M] INFO: PID of FTL process: 143
2024-04-01 00:19:34 2024-04-01 00:19:33.283 [143M] INFO: listening on 0.0.0.0 port 53
2024-04-01 00:19:34 2024-04-01 00:19:33.283 [143M] INFO: listening on :: port 53
2024-04-01 00:19:34 2024-04-01 00:19:33.284 [143M] INFO: PID of FTL process: 143
2024-04-01 00:19:34 2024-04-01 00:19:33.291 [143M] ERROR: SQLite3: recovered 113 frames from WAL file /etc/pihole/pihole-FTL.db-wal (283)
2024-04-01 00:19:34 2024-04-01 00:19:33.291 [143M] INFO: Database version is 17
2024-04-01 00:19:34 2024-04-01 00:19:33.291 [143M] DEBUG_GC: Statvfs() results for /dev/shm:
2024-04-01 00:19:34 2024-04-01 00:19:33.291 [143M] DEBUG_GC: Block size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.291 [143M] DEBUG_GC: Fragment size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.291 [143M] DEBUG_GC: Total blocks: 16384
2024-04-01 00:19:34 2024-04-01 00:19:33.291 [143M] DEBUG_GC: Free blocks: 16127
2024-04-01 00:19:34 2024-04-01 00:19:33.291 [143M] DEBUG_GC: Available blocks: 16127
2024-04-01 00:19:34 2024-04-01 00:19:33.291 [143M] DEBUG_GC: Total inodes: 1004892
2024-04-01 00:19:34 2024-04-01 00:19:33.291 [143M] DEBUG_GC: Free inodes: 1004879
2024-04-01 00:19:34 2024-04-01 00:19:33.291 [143M] DEBUG_GC: Available inodes: 1004879
2024-04-01 00:19:34 2024-04-01 00:19:33.291 [143M] DEBUG_GC: Filesystem ID: 135079855
2024-04-01 00:19:34 2024-04-01 00:19:33.291 [143M] DEBUG_GC: Mount flags: 4142
2024-04-01 00:19:34 2024-04-01 00:19:33.291 [143M] DEBUG_GC: Maximum filename length: 255
2024-04-01 00:19:34 2024-04-01 00:19:33.293 [143M] INFO: Database successfully initialized
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143M] INFO: Imported 0 queries from the on-disk database (it has 0 rows)
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143M] INFO: Parsing queries in database
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143M] INFO: Imported 0 queries from the long-term database
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143M] INFO: -> Total DNS queries: 0
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143M] INFO: -> Cached DNS queries: 0
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143M] INFO: -> Forwarded DNS queries: 0
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143M] INFO: -> Blocked DNS queries: 0
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143M] INFO: -> Unknown DNS queries: 0
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143M] INFO: -> Unique domains: 0
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143M] INFO: -> Unique clients: 0
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143M] INFO: -> DNS cache records: 0
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143M] INFO: -> Known forward destinations: 0
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143M] INFO: FTL is running as user pihole (UID 100)
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Statvfs() results for /etc/pihole/pihole-FTL.db:
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Block size: 1048576
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Fragment size: 4096
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Total blocks: 120699413
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Free blocks: 4289115
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Available blocks: 4289115
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Total inodes: 174876772
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Free inodes: 171564600
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Available inodes: 171564600
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Filesystem ID: 0
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Mount flags: 4134
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Maximum filename length: 255
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Checking free space at /etc/pihole/pihole-FTL.db: 96% > 90%
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143M] INFO: Reading certificate from /etc/pihole/tls.pem ...
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Disk details for path "/etc/pihole/pihole-FTL.db":
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Device or server for filesystem: /host_mark/Users
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Directory mounted on: /etc/pihole
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Type of filesystem: fakeowner
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Comma-separated options for fs: rw,nosuid,nodev,relatime,fakeowner
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Dump frequency (in days): 0
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] DEBUG_GC: Pass number for `fsck': 0
2024-04-01 00:19:34 2024-04-01 00:19:33.297 [143/T145] WARNING: Disk shortage ahead: 96% is used (476.8GB used, 494.4GB total) on fakeowner filesystem mounted at /etc/pihole
2024-04-01 00:19:34 2024-04-01 00:19:33.298 [143M] INFO: Using SSL/TLS certificate file /etc/pihole/tls.pem
2024-04-01 00:19:34 2024-04-01 00:19:33.298 [143/T145] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-04-01 00:19:34 2024-04-01 00:19:33.298 [143/T145] INFO: ----------------------------> FTL crashed! <----------------------------
2024-04-01 00:19:34 2024-04-01 00:19:33.298 [143/T145] INFO: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2024-04-01 00:19:34 2024-04-01 00:19:33.298 [143/T145] INFO: Please report a bug at https://github.com/pi-hole/FTL/issues
2024-04-01 00:19:34 2024-04-01 00:19:33.298 [143/T145] INFO: and include in your report already the following details:
2024-04-01 00:19:34 2024-04-01 00:19:33.298 [143/T145] INFO: FTL has been running for 0 seconds
2024-04-01 00:19:34 2024-04-01 00:19:33.298 [143/T145] INFO: FTL branch: fix/check_disk
2024-04-01 00:19:34 2024-04-01 00:19:33.298 [143/T145] INFO: FTL version: vDev-90dda14
2024-04-01 00:19:34 2024-04-01 00:19:33.298 [143/T145] INFO: FTL commit: 90dda142
2024-04-01 00:19:34 2024-04-01 00:19:33.298 [143/T145] INFO: FTL date: 2024-04-01 09:11:06 +0200
2024-04-01 00:19:34 2024-04-01 00:19:33.298 [143/T145] INFO: FTL user: started as pihole, ended as pihole
2024-04-01 00:19:34 2024-04-01 00:19:33.298 [143/T145] INFO: Compiled for linux/arm64/v8 (compiled on CI) using cc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924
2024-04-01 00:19:34 2024-04-01 00:19:33.298 [143/T145] INFO: Process details: MID: 143
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: PID: 143
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143M] INFO: Restored 0 API sessions from the database
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: TID: 145
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: Name: housekeeper
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: Received signal: Segmentation fault
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: at address: 0x1
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: with code: SEGV_MAPERR (Address not mapped to object)
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: !!! INFO: pihole-FTL has not been compiled with glibc/backtrace support, not generating one !!!
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: ------ Listing content of directory /dev/shm ------
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: File Mode User:Group Size Filename
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: rwxrwxrwx root:root 280 .
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: rwxr-xr-x root:root 340 ..
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: rw------- pihole:pihole 560K FTL-fifo-log
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: rw------- pihole:pihole 4K FTL-per-client-regex
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: rw------- pihole:pihole 12K FTL-dns-cache
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: rw------- pihole:pihole 8K FTL-overTime
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: rw------- pihole:pihole 295K FTL-queries
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: rw------- pihole:pihole 29K FTL-upstreams
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: rw------- pihole:pihole 86K FTL-clients
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: rw------- pihole:pihole 4K FTL-domains
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: rw------- pihole:pihole 82K FTL-strings
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: rw------- pihole:pihole 16 FTL-settings
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: rw------- pihole:pihole 292 FTL-counters
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143M] INFO: Blocking status is enabled
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: rw------- pihole:pihole 88 FTL-lock
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: ---------------------------------------------------
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: Please also include some lines from above the !!!!!!!!! header.
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: Thank you for helping us to improve our FTL engine!
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: Waiting for threads to join
2024-04-01 00:19:34 2024-04-01 00:19:33.299 [143/T145] INFO: Thread database (0) is idle, terminating it.
2024-04-01 00:19:36 2024-04-01 00:19:35.301 [143/T146] INFO: Terminating resolver thread
2024-04-01 00:19:36 2024-04-01 00:19:35.303 [143/T145] INFO: Thread housekeeper (1) is still busy, cancelling it.
2024-04-01 00:19:36 2024-04-01 00:19:35.303 [143/T145] INFO: All threads joined
Okay, one bug (maybe?) fixed, but another one created - or rather: dug out again, I'm pretty sure it's the same one with which we started this thread.
96% is used (476.8GB used, 494.4GB total)
Does that make sense for you on your system? Looks pretty full.
I think I found the original bug and - the only reason I can come up with why it hasn't been found earlier can be that you are the first one trying v6 with an disk filled to the brim! Already fixed as well upstream.
Does that make sense for you on your system? Looks pretty full.
Ha, yes thats accurate 😅. Excuse me while I go clean out some cobwebs...
Ps. Can confirm, cleaned out 46GB of HD screensavers (wow Apple) among other things and indeed, no crash =)
Closing as issue is now fixed
Versions
Core Version is 206cf9c (Latest: null) Branch is development-v6 Hash is 206cf9c4 (Latest: c7d60295) Web Version is ab87e11 (Latest: null) Branch is development-v6 Hash is ab87e110 (Latest: ab87e110) FTL Version is vDev-8e9b1dc (Latest: null) Branch is development-v6 Hash is 8e9b1dc7 (Latest: 9dcdfa86)
Platform
Expected behavior
v6 docker image starts successfully
Actual behavior / bug
I get immediate seg fault. Only thing I noticed is the
cannot read directory '/etc/pihole': Permission denied
which doesnt really make sense to me since Im start with a totally empty parent dir and letting pihole create everything (also it does successfully create all the files, .toml etc). I tried messing with PID/GUID for that, no change in behavior.Steps to reproduce
Steps to reproduce the behavior:
Debug Token
Apologies if Im missing something obvious. Thank you for all your work! 🙏