Closed GvY85 closed 2 years ago
MichaIng
commented
2 years ago Many thanks for your report. Could you do the same monitoring of journalctl -f, to additionally catch all system and service logs? Currently, while there are some entries from the WireGuard module, these look like normal client connections and I don't think it is related. Pi-hole seems to be not related, the database cache resizing is pretty normal.
GvY85
commented
2 years ago Thanks for your reply
The right top screen is journalctl -f.
The wireguard output is a manual debug output I did. I did a reboot so ill try to capture more of the journalctl -f output if it crashes again.
I am supprised by the left bottom screen, any idea what that might be?
MichaIng
commented
2 years ago Not sure either. Is there a kernel upgrade available?
apt update
apt full-upgradeThe system is fully up to date. Now that I look at it it is strange that the journalctl -f output only shows wireguard messages. At least ones a minute there should be a cron job there as well so something went wrong earlier than the log shows,
I have restarted the logging and increased the netrate limit so now ill wait for it to crash again and save the SSH output to text and post it here.
I have done 12hr long memtester runs on free memory and 60min of CPU stresstesting from DietPi so I think its more SW than HW.
Ill post an update when it crashes again.
MichaIng
commented
2 years ago The cron daemon is running?
systemctl status cron
# or
journalctl -u cronI mean at the time of the crash the Cronjob was not running, it is running when the system is up and running. So that is an indicate that something 'off screen' had been going wrong. Lets see what happens when it crashes again
GvY85
commented
2 years ago Ok, got some new crashes:
CRASH #1:
dmesg output (cpu crash at end?):
CRASH #2
dmesg output (rclone crash?):
Joulinar
commented
2 years ago Is this by purpose to block DNS request from your network to Google DNS?
GvY85
commented
2 years ago Yes it is, in my router i have set requests to 8.8.8.8 and 8.8.4.4 to redirect to my pihole device to preven ChromeCast and other IOT devices from bypassing pihole. Not that the pihole handles the traffic this way by looking at the UFW blocks but at least at some point the devices 'give up' and start using pihole according to pihole logs.
It is all very wierd behaviour. Sometimes the system runs for weeks until I manually restart it for some reason and sometimes it crashes very frequently (after few minutes, couple times a day). This week I was doing backups (pc to this server via duplicati and then to some external server with rclone) and I had reboots and crashes so I suspected that it crashes under heavy or peak loads. Did CPU and memory tests but no errors at all. Then I saw some sporatic r8169 device link up/down messages so I suspected the R8111 Ethernet device driver. Changed that to the closed source r8168-dkms driver but no change. Then I changed it back to the stock r8169 driver, reset my bios (in despair) and then I got very frequent link up link down messages. Gave all the ethernet cables a nudge, rebooted the system and now it has been doing nonstop backups (112MB/sec traffic up down over ethernet via external SSD) non stop for 12 hours without a crash or error message :S..... You would allmost think there was a bios setting gone "corrupt" that needed resetting (have not touched bios in years until resetting it yesterday) or swapping to another firmware and back on the realtek r8111 changed some NV setting in the chip or I had bad ethernet connections (physical) that made the system hang (that should not happen). Sry its all very vague but thats all I got.
GvY85
commented
2 years ago I always get the same dmesg errors during boot:
root@PiServer:~# dmesg -T -l emerg,alert,crit,err
[Sat Oct 16 20:16:52 2021] scsi 2:0:0:1: Wrong diagnostic page; asked for 1 got 8
[Sat Oct 16 20:16:52 2021] scsi 2:0:0:1: Failed to get diagnostic page 0x1
[Sat Oct 16 20:16:52 2021] scsi 2:0:0:1: Failed to bind enclosure -19
[Sat Oct 16 20:16:52 2021] EDAC pnd2: Failed to register device with error -22.
[Sat Oct 16 20:16:52 2021] EDAC pnd2: Failed to register device with error -22.
[Sat Oct 16 20:16:52 2021] EDAC pnd2: Failed to register device with error -22.
[Sat Oct 16 20:16:52 2021] EDAC pnd2: Failed to register device with error -22.As far as I know the EDAC messages are due to the CPU supporting ECC but not having ECC memory installed. The scsi messages seem to be related to the external HDD supporting or not supporting some stuff. They are gone when disconnection the drive but the SMART data of the drive seems fine (also replaced the drive some time ago).
MichaIng
commented
2 years ago What is the checkresetnetwork.sh cron job doing every minute? I mean I can imagine what it is about, but could you share the content how it is done?
Also which CPU governor do you use? Could you try to change it, e.g. from schedutil to ondemand, conservative or powersave (to test a very safe option).
CPU temperatures are fine?
G_OBTAIN_CPU_TEMPThe checkresetnetwork.sh pings the gateway (Fritzbox fiber router) every minute and restarts networking when unreachable. I put it in because for some reason the Realtek R8111G does not always reestablish a connection in case of a cable disconnect or when the router reboots. So in case of a power failure or router restart I ended up without Pihole and PiVPN so that could be tricky when on holiday. Contents of the script:
#!/bin/bash
x=$(ping -c1 192.168.178.1 2>&1 | grep Unreachable
echo "$x"
if [ ! "$x"= "" ] then
echo "Network down, attempting restart."
sudo service networking restart
else echo "Network up, nothing to do."
fiGovernor is schedutil and temps idle are 33 degrees and under load 40-50 degrees so nothing strange there. CPU governor options I have are only schedutil and performance and locking all cores to 2500MHz with performance seems a bit wasteful?
I have a gut feeling (looking at al the posts online) it has something to do with my PHY (Realtek R8111G) since it has this strang thing that it wont reconnect automatically and seems to crash when a cable is loose. After I gave all the cables a nudge and restarted the system and defaulted the bios ths system has been up for 36 hours where it was crashing multiple times a day before.
MichaIng
commented
2 years ago sudo service networking restart
That won't restart the network, but only the loopback interface 😉. Did you every observe this functional? To restart the Ethernet interface, do:
ifdown eth0
sleep 1
ifup eth0or the related service:
systemctl restart ifup@eth0But I'd be careful with such automated service restarts: When they are done during the boot sequence, this may cause issues with other services/targets depending on them.
You can skip sudo when just adding that job to the root users crontab, respectively one of the /etc/cron.* directories.
GvY85
commented
2 years ago sudo service networking restart
That won't restart the network, but only the loopback interface 😉. Did you every observe this functional?
Hmm, did not know that :). Wierd thing is, if I restart my fritzbox (that points to this server as DNS server) both my pihole and pivpn clients stay down indefinitely. With this script the device and thus the services come back online after the fritzbox is done rebooting.
The extra sudo is just an oversight :).
MichaIng
commented
2 years ago With this script the device and thus the services come back online after the fritzbox is done rebooting.
Very strange, as this should only bring down and up the loopback interface, nothing more. Or is the eth0 interface defined with auto eth0 in /etc/network/interfaces?
Probably also the loopback interface reload triggers somehow the Ethernet adapter to wake up or so. I mean usually a router restart shouldn't affect the client interface at all, but usually it is immediately usable again as fast as the physical connection has been reestablished. So there seem to be a firmware-wise layer, like a power savings or sleeping mode or so that the connection stays lost 🤔.
GvY85
commented
2 years ago Yes, it is very strange. If I remove the ethernet cable for a few seconds it usually is ok and restores the connection. If I wait longer (say 3-10 sec+) or plug it in to an unmanaged switch instead of directly into the Fritzbox or into another port of the Fritzbox the connection does not restore at all. And like I said, if I reboot the Fritzbox manually from its interface the interface does not come back online as well. Everyone says it should but it doesn't in my case.
eth0 in /etc/network/interfaces is defined as:
auto eth0
iface eth0 inet static
address (ip adress of the server)
netmask 255.255.255.0
gateway (ip adress of the fritzbox)
dns-nameserver 127.0.0.1 Dont know why the DNS server is in there as 127.0.0.1 to be honest, perhaps because of PiHole + Unbound.
Joulinar
commented
2 years ago The real truth for DNS server should be cat /etc/resolv.conf. And I would not recommend to use localhost. This has more downsides, than it will block adds 😉
MichaIng
commented
2 years ago Okay, makes sense then. adress is a typo only here or in the file? Agree with @Joulinar to not use Pi-hole for DNS by the server itself. Keep using a regular upstream provider as long as do no web browser on that server itself 😉.
GvY85
commented
2 years ago adress is a typo indeed :) (sry, logged in on a work pc and ssh to the server from my phone so am typing stuff over).
I am using unbound, so in that case why not have unbound resolve it by pointing to 127.0.0.1? What are the downsides in this case? Have been using it for years like this I think :)
MichaIng
commented
2 years ago E.g. Unbound might be stopped during a package upgrade, and it may then break its own network capability so that the upgrade cannot succeed. Or it may fail for any other reason and you then cannot reinstall the package because it broke its own DNS. It's like maintaining/fixing the branch you're sitting on 😉.
But most importantly, is there any reason why you wouldn't want to use public DNS for the requests that server does? For ad blocking this cannot be, but if you don't want to have any public provider know that your server is contacting e.g. GitHub or dietpi.com to check for updates, or Debian/Raspbian servers for APT updates, then of course you may use Unbound (and manually change /etc/resolv.conf in case of an Unbound failure). But I think this is pretty much an edge case, while for a desktop system with web browser, where you do work, social media, online banking, porn 😄, or whatever, this is a very different topic, of course.
Joulinar
commented
2 years ago Just using localhost will direct your DNS request to Pihole as Pihole is running on port 53. Usually Pihole is the one addressing Unbound, and not clients.
Anyway what will happen if unbound or Pihole are not working? Correct you would need to adjust your local DNS server settings to be able to get back internet access in DietPi. This could be avoided by using a public DNS server directly. As well we had a couple of reports in past where Pihole was blocking some GitHub address. And there shouldn't be anything to block, as long as you don't use a web browser on DietPi. 😉
GvY85
commented
2 years ago Hmm, that does make sense indeed. Never realized you would need to set an external DNS for the PiHole server.
But then, this wouldnt cause these wierd crashes and the PHY not reconnecting right ;)?
MichaIng
commented
2 years ago In the past Pi-hole actually automatically configured itself as DNS, but that got removed for exactly this reason: It usually doesn't make sense and causes more issues than benefits. As long as you know how to change it manually when Pi-hole/Unbound fails, all is fine, but usually no need to risk that situation 😉.
MichaIng
commented
2 years ago I lost track here. Is there still an issue/question remaining?
GvY85
commented
2 years ago Sry bit late but no there is not. Did a bios update, switched the bios to ACPI mode to test and since then no crashes anymore. Still have to run this script everyfew minutes (strange) in order to get the ethernet nic to reconnect if the router reboots or the cable gets disconnected and reinserted but that is about it.
#!/bin/bash
x=$(ping -c1 192.168.178.1 2>&1 | grep Unreachable
echo "$x"
if [ ! "$x"= "" ] then
echo "Network down, attempting restart."
sudo service networking restart
else echo "Network up, nothing to do."
fiOkay great. That the Ethernet connection is not usable after a temporary router reboot is indeed strange. Probably a rare hardware specific limitation. However, good that you have a workaround. Probably we will widen the DietPi-WiFi-Monitor do be a general connectivity monitor to either restart the current connection (like it does now and your script does) or alternatively fallback to another adapter until the preferred/main adapter connectivity is up again: #3254
Creating a bug report/issue
Required Information
I am having random crashes on my Pihole + PiVPN + Unbound system. I thought the problem was Pihole so I finally managed to log it over SSH but now I am not so sure what the problem is. I am using a Watchdog and a Kernel Panic counter so my system reboots itself most of the times but there definately is a problem. Is this hardware related?