Closed udochrist closed 3 years ago
EDT: Changing between PIHOLEDNS and DNS1 DNS2 vars in the config between these 2 tries in the docker-compose file
waiting 10 mins after the last try and restarting
docker-compose up pihole
now logs
and seems to repeat the "starting" "Stopping" forever
Can you please add the environment variable PH_VERBOSE
and set it to 1
and then startup the container?
here you are. thanks for picking this up so fast...
Hmm, that hasn't given me anything useful 😔
Have you explicitly run docker-compose down
any time before updating? My standard process on my Synology is:
docker-compose pull && docker-compose down && docker-compose up -d
What does your compose file look like?
Also, see here for a note on the capabilities messages you were seeing
1/ yes
I have just managed to exec a bash into the container while it seems to starting/crashing the pihole-ftl process. the pihole-FTL.log shows:
[2021-10-24 11:20:32.537 645M] Resizing "FTL-queries" from 59408384 to (1064960 * 56) == 59637760 (/dev/shm: 59.9MB used, 67.1MB total, FTL uses 59.9MB)
[2021-10-24 11:20:32.543 645M] Resizing "FTL-queries" from 59637760 to (1069056 * 56) == 59867136 (/dev/shm: 60.1MB used, 67.1MB total, FTL uses 60.1MB)
[2021-10-24 11:20:32.549 645M] Resizing "FTL-queries" from 59867136 to (1073152 * 56) == 60096512 (/dev/shm: 60.4MB used, 67.1MB total, FTL uses 60.4MB)
[2021-10-24 11:20:32.555 645M] Resizing "FTL-queries" from 60096512 to (1077248 * 56) == 60325888 (/dev/shm: 60.6MB used, 67.1MB total, FTL uses 60.6MB)
[2021-10-24 11:20:32.561 645M] Resizing "FTL-queries" from 60325888 to (1081344 * 56) == 60555264 (/dev/shm: 60.8MB used, 67.1MB total, FTL uses 60.8MB)
[2021-10-24 11:20:32.568 645M] Resizing "FTL-queries" from 60555264 to (1085440 * 56) == 60784640 (/dev/shm: 61.1MB used, 67.1MB total, FTL uses 61.0MB)
[2021-10-24 11:20:32.574 645M] Resizing "FTL-queries" from 60784640 to (1089536 * 56) == 61014016 (/dev/shm: 61.3MB used, 67.1MB total, FTL uses 61.3MB)
[2021-10-24 11:20:32.574 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.581 645M] Resizing "FTL-queries" from 61014016 to (1093632 * 56) == 61243392 (/dev/shm: 61.5MB used, 67.1MB total, FTL uses 61.5MB)
[2021-10-24 11:20:32.581 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.587 645M] Resizing "FTL-queries" from 61243392 to (1097728 * 56) == 61472768 (/dev/shm: 61.7MB used, 67.1MB total, FTL uses 61.7MB)
[2021-10-24 11:20:32.588 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.594 645M] Resizing "FTL-queries" from 61472768 to (1101824 * 56) == 61702144 (/dev/shm: 62.0MB used, 67.1MB total, FTL uses 62.0MB)
[2021-10-24 11:20:32.594 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.600 645M] Resizing "FTL-queries" from 61702144 to (1105920 * 56) == 61931520 (/dev/shm: 62.2MB used, 67.1MB total, FTL uses 62.2MB)
[2021-10-24 11:20:32.600 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.606 645M] Resizing "FTL-queries" from 61931520 to (1110016 * 56) == 62160896 (/dev/shm: 62.4MB used, 67.1MB total, FTL uses 62.4MB)
[2021-10-24 11:20:32.606 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.612 645M] Resizing "FTL-queries" from 62160896 to (1114112 * 56) == 62390272 (/dev/shm: 62.7MB used, 67.1MB total, FTL uses 62.6MB)
[2021-10-24 11:20:32.612 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.618 645M] Resizing "FTL-queries" from 62390272 to (1118208 * 56) == 62619648 (/dev/shm: 62.9MB used, 67.1MB total, FTL uses 62.9MB)
[2021-10-24 11:20:32.618 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.624 645M] Resizing "FTL-queries" from 62619648 to (1122304 * 56) == 62849024 (/dev/shm: 63.1MB used, 67.1MB total, FTL uses 63.1MB)
[2021-10-24 11:20:32.624 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.630 645M] Resizing "FTL-queries" from 62849024 to (1126400 * 56) == 63078400 (/dev/shm: 63.3MB used, 67.1MB total, FTL uses 63.3MB)
[2021-10-24 11:20:32.630 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.636 645M] Resizing "FTL-queries" from 63078400 to (1130496 * 56) == 63307776 (/dev/shm: 63.6MB used, 67.1MB total, FTL uses 63.6MB)
[2021-10-24 11:20:32.636 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.642 645M] Resizing "FTL-queries" from 63307776 to (1134592 * 56) == 63537152 (/dev/shm: 63.8MB used, 67.1MB total, FTL uses 63.8MB)
[2021-10-24 11:20:32.642 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.648 645M] Resizing "FTL-queries" from 63537152 to (1138688 * 56) == 63766528 (/dev/shm: 64.0MB used, 67.1MB total, FTL uses 64.0MB)
[2021-10-24 11:20:32.648 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.655 645M] Resizing "FTL-queries" from 63766528 to (1142784 * 56) == 63995904 (/dev/shm: 64.3MB used, 67.1MB total, FTL uses 64.3MB)
[2021-10-24 11:20:32.655 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.661 645M] Resizing "FTL-queries" from 63995904 to (1146880 * 56) == 64225280 (/dev/shm: 64.5MB used, 67.1MB total, FTL uses 64.5MB)
[2021-10-24 11:20:32.661 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.667 645M] Resizing "FTL-queries" from 64225280 to (1150976 * 56) == 64454656 (/dev/shm: 64.7MB used, 67.1MB total, FTL uses 64.7MB)
[2021-10-24 11:20:32.667 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.673 645M] Resizing "FTL-queries" from 64454656 to (1155072 * 56) == 64684032 (/dev/shm: 65.0MB used, 67.1MB total, FTL uses 64.9MB)
[2021-10-24 11:20:32.673 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.679 645M] Resizing "FTL-queries" from 64684032 to (1159168 * 56) == 64913408 (/dev/shm: 65.2MB used, 67.1MB total, FTL uses 65.2MB)
[2021-10-24 11:20:32.679 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.685 645M] Resizing "FTL-queries" from 64913408 to (1163264 * 56) == 65142784 (/dev/shm: 65.4MB used, 67.1MB total, FTL uses 65.4MB)
[2021-10-24 11:20:32.685 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.692 645M] Resizing "FTL-queries" from 65142784 to (1167360 * 56) == 65372160 (/dev/shm: 65.6MB used, 67.1MB total, FTL uses 65.6MB)
[2021-10-24 11:20:32.692 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.698 645M] Resizing "FTL-queries" from 65372160 to (1171456 * 56) == 65601536 (/dev/shm: 65.9MB used, 67.1MB total, FTL uses 65.9MB)
[2021-10-24 11:20:32.698 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.704 645M] Resizing "FTL-queries" from 65601536 to (1175552 * 56) == 65830912 (/dev/shm: 66.1MB used, 67.1MB total, FTL uses 66.1MB)
[2021-10-24 11:20:32.704 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.710 645M] Resizing "FTL-queries" from 65830912 to (1179648 * 56) == 66060288 (/dev/shm: 66.3MB used, 67.1MB total, FTL uses 66.3MB)
[2021-10-24 11:20:32.710 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.717 645M] Resizing "FTL-queries" from 66060288 to (1183744 * 56) == 66289664 (/dev/shm: 66.6MB used, 67.1MB total, FTL uses 66.5MB)
[2021-10-24 11:20:32.717 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.723 645M] Resizing "FTL-queries" from 66289664 to (1187840 * 56) == 66519040 (/dev/shm: 66.8MB used, 67.1MB total, FTL uses 66.8MB)
[2021-10-24 11:20:32.723 645M] WARNING: More than 90% of /dev/shm is used
[2021-10-24 11:20:32.726 645M] New upstream server: 1.1.1.1:53 (2/32)
[2021-10-24 11:20:32.727 645M] New upstream server: 1.0.0.1:53 (3/32)
[2021-10-24 11:20:32.730 645M] Imported 1187522 queries from the long-term database
[2021-10-24 11:20:32.731 645M] -> Total DNS queries: 1187522
[2021-10-24 11:20:32.731 645M] -> Cached DNS queries: 266
[2021-10-24 11:20:32.731 645M] -> Forwarded DNS queries: 1185914
[2021-10-24 11:20:32.731 645M] -> Blocked DNS queries: 28
[2021-10-24 11:20:32.731 645M] -> Unknown DNS queries: 0
[2021-10-24 11:20:32.731 645M] -> Unique domains: 413
[2021-10-24 11:20:32.731 645M] -> Unique clients: 29
[2021-10-24 11:20:32.731 645M] -> Known forward destinations: 4
[2021-10-24 11:20:32.731 645M] Successfully accessed setupVars.conf
[2021-10-24 11:20:32.731 645M] WARNING: Required Linux capability CAP_SYS_NICE not available
[2021-10-24 11:20:32.731 645M] WARNING: Required Linux capability CAP_IPC_LOCK not available
[2021-10-24 11:20:32.731 645M] listening on 0.0.0.0 port 53
[2021-10-24 11:20:32.731 645M] listening on :: port 53
[2021-10-24 11:20:32.812 645M] PID of FTL process: 645
[2021-10-24 11:20:32.813 645M] INFO: FTL is running as root
[2021-10-24 11:20:32.813 645/T646] Listening on port 4711 for incoming IPv4 telnet connections
[2021-10-24 11:20:32.813 645/T648] Listening on Unix socket
[2021-10-24 11:20:32.813 645M] Reloading DNS cache
[2021-10-24 11:20:33.896 645/T649] Compiled 0 whitelist and 0 blacklist regex filters for 29 clients in 0.1 msec
[2021-10-24 11:20:34.213 645M] Blocking status is enabled
ist it possible there's not enough ram suddenly for the container? the Synology has plenty left free
[2021-10-24 11:27:59.591 1743M] WARN: Could not fallocate() in realloc_shm() (/__w/FTL/FTL/src/shmem.c:789): No space left on device
[2021-10-24 11:27:59.591 1743M] FATAL: realloc_shm(): Failed to resize "FTL-queries" (18) to 66748416: No space left on device (28)
[2021-10-24 11:27:59.752 1759M] Using log file /var/log/pihole-FTL.log
[2021-10-24 11:27:59.752 1759M] ########## FTL started on pihole! ##########
[2021-10-24 11:27:59.752 1759M] FTL branch: master
[2021-10-24 11:27:59.752 1759M] FTL version: v5.11
and this seems to be the issue... hard to catch in the log
the container says:
Filesystem Size Used Avail Use% Mounted on
none 80T 76T 4.3T 95% /
tmpfs 68M 0 68M 0% /dev
tmpfs 17G 0 17G 0% /sys/fs/cgroup
shm 68M 68M 91k 100% /dev/shm
/dev/vg1000/lv 80T 76T 4.3T 95% /etc/pihole
tmpfs 17G 0 17G 0% /proc/acpi
tmpfs 17G 0 17G 0% /proc/scsi
tmpfs 17G 0 17G 0% /sys/firmware
so the tempfs mounted /dev/shm is out of space. any ideas on how to enlarge that?
I've no idea if this will have any effect (i'm still quite green when it comes to Docker), but you should be able to increase the size of /dev/shm/
in your container like so: https://stackoverflow.com/a/56655091
e.g:
version: '3.5'
services:
your_service:
shm_size: '2gb' <--- Or whatever value you want
I'll have a poke about on my running image and see what I can see
For me, it seems 64M
is enough (currently only using 3.8M)
Filesystem Size Used Avail Use% Mounted on
/dev/md2 6.2T 3.7T 2.5T 60% /
tmpfs 64M 0 64M 0% /dev
tmpfs 4.8G 0 4.8G 0% /sys/fs/cgroup
shm 64M 3.8M 61M 6% /dev/shm
/dev/md2 6.2T 3.7T 2.5T 60% /etc/pihole
/dev/md2 6.2T 3.7T 2.5T 60% /etc/hosts
tmpfs 4.8G 0 4.8G 0% /proc/acpi
tmpfs 4.8G 0 4.8G 0% /proc/scsi
tmpfs 4.8G 0 4.8G 0% /sys/firmware
@DL6ER is there anything you can see from the information already available that would cause FTL memory usage to be high in this user's case?
YES!! that did it. I went to 512MB and it s now working again!
it was 3M too small before :-)
Filesystem Size Used Avail Use% Mounted on
none 80T 76T 4.3T 95% /
tmpfs 68M 0 68M 0% /dev
tmpfs 17G 0 17G 0% /sys/fs/cgroup
shm 537M 68M 470M 13% /dev/shm
/dev/vg1000/lv 80T 76T 4.3T 95% /etc/pihole
tmpfs 17G 0 17G 0% /proc/acpi
tmpfs 17G 0 17G 0% /proc/scsi
tmpfs 17G 0 17G 0% /sys/firmware
Now that you're up and running, can we get a debug token please?
of course.
here you are: https://tricorder.pi-hole.net/VPi9Nh9h/
192.168.1.1
is hitting FTL's default rate limit of 1000 queries in 60 seconds...
I wonder if you have some kind of feedback loop going on, where Pi-hole is querying 192.168.1.1
, and in turn that is then sending the query back to pi-hole
If you are using Pi-hole as your dhcp server, do you still need the conditional forwarding set up?
I‘ll re-check the config.
Pi holes dhcp Server is not enabled in my config. The dhcp Server is 192.168.1.1 (which is using pi.hole to resolve dns). i think I added the forwarding quite some time ago to see the client names in the pi-hole gui
Ah, yes, sorry I missed the DHCP_ACTIVE=false
, I was going by the existence of 04-pihole-static-dhcp.conf
and some of the other settings in setupVars.conf
, in which case that's a perfectly valid reason to have conditional forwarding turned on!
I guess if you have clients that are also querying 192.168.1.1
for DNS, then those queries will be passed onto the Pi-hole.
Anyway, The main issue is resolved - but I would certainly recommend going over your config to look for any misconfigurations !
is there anything you can see from the information already available that would cause FTL memory usage to be high in this user's case?
Yes, it is what you have found, too, FTL actually logged a warning that there is trouble ahead:
[2021-10-24 11:20:32.723 645M] Resizing "FTL-queries" from 66289664 to (1187840 * 56) == 66519040 (/dev/shm: 66.8MB used, 67.1MB total, FTL uses 66.8MB) [2021-10-24 11:20:32.723 645M] WARNING: More than 90% of /dev/shm is used
eventually leading to something like
[2021-10-24 11:27:59.591 1743M] WARN: Could not fallocate() in realloc_shm() (/__w/FTL/FTL/src/shmem.c:789): No space left on device [2021-10-24 11:27:59.591 1743M] FATAL: realloc_shm(): Failed to resize "FTL-queries" (18) to 66748416: No space left on device (28)
At this point 1,187,840 queries have been imported into memory. Even when Pi-hole is really efficient on memory: over one million queries is not expected to work with only 64MB of RAM, you need more. Mind that the space in /dev/shm
is nothing else than RAM materialized into files. It has nothing to do with SD card disk space.
Versions
cannot run any command as the docker container exits itself
Platform
Expected behavior
The same config of docker-compose is running since 2 years and updating to the latest image has never caused any issues.
has always started and will always update and start. Further expectancy would be a meaningful error log to identify the issue.
Actual behavior / bug
after a pull to pi-hole/pi-hole:latest for my docker install of pihole the container is no longer starting up. docker-compose shows it as "exiting".
The errors I can identify in the log output are
and
Steps to reproduce
Steps to reproduce the behavior:
docker-compose pull && docker-compose up -d (in this case docker-compose up to see the logs and find out why its crashing
Debug Token
Screenshots
output from docker-compose up pihole
Additional context
Add any other context about the problem here.