Speedtest testing schedule no longer kicking off script

[ChrisWNY]

For about 2 days now I've noticed the speedtest testing schedule is no longer kicking off the speedtest script every 12 hours. It would consistently run at 00:00 and 12:00 daily, the past 2 days it just hasn't triggered for whatever reason. I understand I could set up a cron job to just run the script every 12 hours, but that might cause some issues if and when the scheduled job started running on its own again.

What can I check to verify the scheduled job runs as expected? Checking the "run test now" works just fine from the Speedtest UI, as well as manually executing the script from the CLI. Just confused as to why it stopped running automatically (seems to coincide with when I rebooted my Rasp Pi after an update a few days ago).

[arevindh]

@ChrisWNY @ipitio I also had a similar experience, sometimes it stops running for no reason.

[ChrisWNY]

Did you end up just adding a cron job to work around the issue?

0 */12 * * * sudo "/var/www/html/admin/scripts/pi-hole/speedtest/speedtest.sh""

[ipitio]

Can you guys keep an eye on pihole-speedtest.timer with systemctl list-timers --all or systemctl status pihole-speedtest.timer

[ChrisWNY]

Looks like my timer wasn't running, at least based on this output:

`systemctl status pihole-speedtest.timer pihole-speedtest.timer - Pi-hole Speedtest Timer Loaded: loaded (/etc/systemd/system/pihole-speedtest.timer; static) Active: active (waiting) since Mon 2023-02-27 12:02:03 EST; 22min ago Trigger: Tue 2023-02-28 00:00:00 EST; 11h left Triggers: pihole-speedtest.service

Feb 27 12:02:03 pihole systemd[1]: Started Pi-hole Speedtest Timer`

[ipitio]

Well it's running now.. what changed 22 min ago

[ChrisWNY]

I ran the speedtest "run test now" manually from the web UI around that time.

[arevindh]

Did you end up just adding a cron job to work around the issue?

0 */12 * * * sudo "/var/www/html/admin/scripts/pi-hole/speedtest/speedtest.sh""

I tested from UI and started working again, no idea what is the issue.

[ipitio]

Can you guys keep an eye on pihole-speedtest.timer with systemctl list-timers --all or systemctl status pihole-speedtest.timer

Using the list command you can see the next and last times it's supposed to activate the service. If it says it worked but it didn't, that could be a clue

[ChrisWNY]

Apologies if this doesn't display correctly...it looks like the speedtest timer has properly set up the next job to run (at 00:00 Feb 28)...

NEXT LEFT LAST PASSED UNIT ACTIVATES Mon 2023-02-27 12:39:00 EST 12s left Mon 2023-02-27 12:09:02 EST 29min ago phpsessionclean.timer phpsessionclean.service Mon 2023-02-27 14:58:08 EST 2h 19min left Mon 2023-02-27 00:03:10 EST 12h ago certbot.timer certbot.service Mon 2023-02-27 22:54:44 EST 10h left Mon 2023-02-27 09:25:40 EST 3h 13min ago apt-daily.timer apt-daily.service Mon 2023-02-27 23:55:00 EST 11h left Sun 2023-02-26 23:55:11 EST 12h ago log2ram-daily.timer log2ram-daily.service Tue 2023-02-28 00:00:00 EST 11h left Mon 2023-02-27 00:00:47 EST 12h ago logrotate.timer logrotate.service Tue 2023-02-28 00:00:00 EST 11h left Mon 2023-02-27 00:00:47 EST 12h ago man-db.timer man-db.service Tue 2023-02-28 00:00:00 EST 11h left n/a n/a pihole-speedtest.timer pihole-speedtest.service Tue 2023-02-28 00:25:45 EST 11h left Mon 2023-02-27 00:25:45 EST 12h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service Tue 2023-02-28 06:31:37 EST 17h left Mon 2023-02-27 06:52:40 EST 5h 46min ago apt-daily-upgrade.timer apt-daily-upgrade.service Sun 2023-03-05 03:10:01 EST 5 days left Sun 2023-02-26 03:10:19 EST 1 day 9h ago e2scrub_all.timer e2scrub_all.service Mon 2023-03-06 01:30:01 EST 6 days left Mon 2023-02-27 00:03:17 EST 12h ago fstrim.timer fstrim.service

[ipitio]

Hmm so it stops working after a reboot then starts again after running a test now. Yep peculiar indeed. Could you keep an eye on that list and let me know what it says when it stops working and what you just did

[ChrisWNY]

Btw, is this regenerated after a reboot?

sudo systemctl edit --full pihole-speedtest.service

I noticed the CPU quota was back to 20% when the script kicked off at 00:00 last night. On the upside, the script did run automatically.

[ipitio]

The only way that should be possible is if the mod is reinstalled at some point

[ChrisWNY]

Something is resetting the config...haven't run a reinstall, but just checked again and the CPU quota 20% was back in there. I'll keep an eye on the file to find out when it's being modified (might be after the script is executed?)...the file I'm referring to is in

/etc/systemd/system/pihole-speedtest.service

[ipitio]

Oh, the service is also reset whenever changing the schedule in admin

[ChrisWNY]

Yeah that might've been what did it...though I know I got rid of the CPU quota statement some time after the automatic script run at 00:00 last night, and had to re-edit that file again this morning.

[ipitio]

That is strange. Make sure to run daemon-reload after the edit. In any case, it shouldn't be a problem once sid merges the change to remove the quota

[arevindh]

That is strange. Make sure to run daemon-reload after the edit. In any case, it shouldn't be a problem once sid merges the change to remove the quota

This one was already removed

https://github.com/arevindh/pi-hole/commit/523297968e501724ab9725cde04c1113e0f2a30b

[ipitio]

Oh, @ChrisWNY update now and it shouldn't happen again.

[ChrisWNY]

Will do...should I update using the web UI or via the manual update method?

[ipitio]

UI should be fine if you already have the speedtest source, you might have to run pihole restartdns at the end though, until the latest PR to this repo is merged. You'll also need to wail for this PR if you don't have the source

[ChrisWNY]

I can confirm that after rebooting my Pi the speedtest timer ends up inactive/dead. What gets it started again is kicking off the "run test now" from Settings the timer goes back to normal. Confirmed it because of a Pi OS update last night that required a reboot.

[ipitio]

If you saved the output before running run test now, like here, could you paste it again? I don't know why run test now would start the timer because all it does is directly run speedtest.sh. RTN fixes the timer because the timer is reenabled every time settings are saved, so for some reason your pi doesn't like the wantedby timers.target maybe? Try changing it to default.target and the others and see which one works.

[j-broo]

I have the same problem. Doing a "Run Test Now" works, then subsequent tests seem to follow hourly, but my Pi reboots nightly on a schedule, so tests basically never happen after the first reboot, post manual test. Any solution? I've been checking regularly for updates here but nothing since the last one on 5 Jan 2023.

[ipitio]

@ChrisWNY can you try updating with the script on my branch here https://github.com/ipitio/pihole-speedtest/raw/ipitio/mod.sh and see if the timer works after a reboot. You'll need to save/confirm Speedtest settings in admin once after updating, before rebooting.

[ChrisWNY]

Sure - will have that tested shortly. I recently migrated my Pi-hole installation to a x86 mini-PC running Ubuntu, the Pi 4b was starting to flake out, so I went with something more robust.

Will report back shortly.

[ChrisWNY]

Noticed when running the script it throws an error on line 229, "expected ;"

[ipitio]

Are you using a recent version of bash?

https://stackoverflow.com/a/24544780

[ChrisWNY]

I ran it through Bash...script starts running but stops when it can't identify the OS (I'm running Pi-hole on Ubuntu 22.x).

Are there some quick manual steps I can run to grab the repo/script that corrects the timer after a reboot? I viewed the contents of the script and can see where it's pulling the repo from...looks like there are several other components being updated within the script (pi-hole, the speedtest CLI, etc.).

[ipitio]

Yeah, if you look at my PR linked above, it shows which lines to move in pimod.timer. Then you have to daemon-reload and reenable like in webpage.sh. Though that script will overwrite with the old version whenever you save settings.

It fails to identify Ubuntu? I'll have to look into that.

[ChrisWNY]

The script output the following:

Installing /etc/apt/sources.list.d/ookla_speedtest-cli.list...curl: (22) The requested URL returned error: 404

Unable to download repo config from: https://packagecloud.io/install/repositories/ookla/speedtest-cli/config_file.list?os=debian&dist=jammy&source=script

This usually happens if your operating system is not supported by packagecloud.io, or this script's OS detection failed.

You can override the OS detection by setting os= and dist= prior to running this script. You can find a list of supported OSes and distributions on our website: https://packagecloud.io/docs#os_distro_version

For example, to force Ubuntu Trusty: os=ubuntu dist=trusty ./script.sh

If you are running a supported OS, please email support@packagecloud.io and report this. Mon Mar 20 10:42:42 PM EDT 2023 - Process Aborted Mon Mar 20 10:42:42 PM EDT 2023 - A restore is not needed or one failed. Mon Mar 20 10:42:42 PM EDT 2023 - Please try again or try manually.

It's not so much the script but one of the packages it tried to install, I think.

[ipitio]

Oh it's detecting Ubuntu as Debian. Will fix!

[arevindh]

@ipitio can't we use the script provided without us checking for the os ?

[ChrisWNY]

I made the changes to the webpage.sh script as mentioned in the PR, down to the exact line numbers, everything matches up. Rebooted and the timer does appear to persist post-reboot.

[ipitio]

@ipitio can't we use the script provided without us checking for the os ?

Yeah we could, but iirc it detected raspbian as Debian even though they have package for it and this is in their script:

[town3r]

the Pi 4b was starting to flake out

How so? Was it still utilizing an sd card? I switched over to a M2 SATA SSD via this and it's been running great as far as I can tell.

[ChrisWNY]

the Pi 4b was starting to flake out

How so? Was it still utilizing an sd card? I switched over to a M2 SATA SSD via this and it's been running great as far as I can tell.

I am also using a SATA SSD (external USB) but the Pi 4b device itself is erratic...it crashed a couple of times over the past few weeks for no apparent reason, nothing in the logs to go by. I believe it has something to do with volume of requests...CPU usage was steadily increasing and the number of clients hitting it seemed to be about all it could handle. So I replaced it with a x86 mini-PC, Intel N5105 CPU w/16GB of DDR4 RAM and 256GB of internal NVMe storage. Installed Ubuntu 22.x on there and it doesn't even come close to breaking a sweat. Oddly enough I got the whole mini PC with all that storage and RAM (plus dual 1Gbps NICs) for barely more than a Pi 4b would cost. I still have the Pi 4b, but using it as a backup DNS server w/identical Pi-hole config on both. It's been relieved of daily speedtests and it's been stable with lower network volume.

[j-broo]

I'm running a first gen Pi: Raspberry Pi Model B Plus Rev 1.2

Got a Sandisk A1 micro SD card in it.

Load averages: Load: 0.05 0.17 0.24

Also running NUT server on it for my UPSes, as well as Zerotier One. Working just fine.

I think maybe your performance issue lies elsewhere.

[j-broo]

[ChrisWNY]

My load averages are definitely much higher on the Pi 4b, I'll double check them later today. I had over 60 clients pointing at it, including a number of IoT devices that were somewhat "noisy." The x86 box literally has load averages barely above 0, the Pi was often up over 1.x especially during the daytime hours.

[j-broo]

Was the Pi on ethernet? Mine was in WiFi and just recently I changed it to ethernet (i moved it closer to the switch) and my load averages plummeted from around the 1.0 mark and more to the stats above.

[ChrisWNY]

It was on ethernet...Wifi has always been disabled. I run a wire to anything that uses a static IP or hosts any client/server app on my home LAN. My Pi 4b is roughly 5-6 yrs old now. My Pi has a RAM disk set up, all logs are written to RAM. I did this back when I used a micro SD card to minimize writes to the card, and just sort of left things that way even after migrating to an external SSD. I wonder if all of that writing back and forth to and from RAM have negatively impacted the Pi's internal RAM lifespan. Anything's possible.