Action when analyzes are stopped

Background: I helped a bird observatory install BirdNET-Pi. The installation is local and connects via radio link to a private network which is then connected to the internet via a dynamic IP address. I can not monitor the installation remotely and it takes several hours to go out there. Daily charts are sent regularly via FTP to a web hotel.

Problem: Twice in two weeks, the analysis of the sounds has stopped working. I can see that the daily graph has not been updated and that data has not been sent to BirdWeather. When the error first occurred, the whole thing was solved with power off / power on by a visitor out there. I suspect, but am not sure, BirdNET-Pi somehow hangs when there is a problem with the network or internet connection and doesn't start working again when connection is back again. The OS itself is working. I can see that FTP works with mirroring the charts directory every hour to my web hotel.

Solution: Would it be possible to build a function that automatically detects if, for example, analyzes are not performed or detections are not sent to BirdWeather and that action is taken, such as restarting services or rebooting? I think more people can benefit from a similar feature when the program is spread and perhaps installed in places where monitoring is not possible from the outside.

I'm seeing something similar. my 3b+ pi is running really hot so I'm waiting to see if adding a fan fixes my issue.

Hi, @Svardsten53

Problem: Twice in two weeks, the analysis of the sounds has stopped working. I can see that the daily graph has not been updated and that data has not been sent to BirdWeather. When the error first occurred, the whole thing was solved with power off / power on by a visitor out there. I suspect, but am not sure, BirdNET-Pi somehow hangs when there is a problem with the network or internet connection and doesn't start working again when connection is back again. The OS itself is working. I can see that FTP works with mirroring the charts directory every hour to my web hotel.

By chance were you able to gather any log excerpts from around the time you think analysis first stopped? Where you able to determine whether the analysis service or the recording service was the issue, (i.e., did data stop getting analyzed or did it stop getting collected)?

I can't say that I agree with the assessment that unexpected network loss causes any type of problem in terms of data collection/processing/analysis, as I haven't seen evidence of network loss hindering the collection and analysis services. Indeed, the recent pre-installed image relies on BirdNET-Pi being able to function without any or with a dynamic networking environment. (It only ever needs access to the loopback interface, which is not affected when connecting/disconnecting to host networks.)

Also, if you're seeing FTP uploads, that does imply connectivity, so we at least know the connectivity, if lost, is coming back.

With more information we ought to be able to implement some tests/checks that can address what we find as the root cause. Until then, I won't be implementing any kind of automatic reboot or blanket service restart without knowing what problem the restart is meant to address.

The best place to go from here is trying to determine if the analysis is stopping because the analysis is stopping, or because the system has stopped collecting data. Then we will know whether the reboot has addressed the analysis service or the recording service.

@mellery Would you be able to provide any specifics about your observations that make you think you're seeing the same behavior @Svardsten53 has described with his unique setup?

@mcguirepr89 I have a normal network setup, but I see analysis stopping too, fixed by a hard reset. Symptoms sound the same as @Svardsten53 , cause could be different. I think my pi is overheating (it's running at 60C+) and cpu is auto throttling to the point where the analysis won't run. I can report back later this when I get a fan installed, or open a separate ticket. Are the running temperatures on @Svardsten53 's pi lower?

The installation is the same as the one I have at home, the same Rpi 4+ and the same software version. The network, on the other hand, is different - a subnet with .birds as suffix instead of .local and I do not know what the connection of the main network to the internet looks like.

Unfortunately I have no opportunities to catch logs as it takes me two hours to go there by car and boat, otherwise it is unmanned most of the time. The only thing anyone else can do is to power off / power on and it will hopefully be done this weekend.

I understand that more facts are needed to make a solution. Next time I go out I will try to make a quick and dirty solution, just to avoid downtime. I'm thinking of putting restart of services in crontab every few hours and on top of that maybe even a reboot every night.

@mellery: My installation at home does not have an ideal air exchange and the temperature is around 70C, but I have never experienced that it has caused any problems. I assume the temperature on the other installation is lower.

This may constitute a different thread but in my case I have a pi-400 with an SSD that just can't keep up with the ingestion of new recordings (I have a plague of Goldfinches in my garden :). The effect of this is that the size of the "files" array built from the RECS_DIR listing becomes too much to process and it will hang on analyzing. Another effect is that the disk will fill up - not always due to number of WAV files but sometimes from the ffmpeg logging filling up the syslog. My fix for this so far has been to:

send stdout and stderr to their own service logs and have a more bespoke and aggressive logrotate
reduce the amount of ffmpeg logging by sending stderr (in birdnet_recording.sh) to a tmp file that is overwritten each time
use a small throttling script that monitors the number of WAV files in RECS_DIR and stops the birdnet_recording service until it drops below the threshold (I set it to 100). This runs from crontab on @reboot.

I'm still finding my way around this great project so there may well be facilities already for doing this.

Here's the throttle script - I have edited this a few times since to include a services restart if "analyzing now" doesn't change for ten checks:

#!/bin/bash
#
# Stop the recordings while the analysis catches up
#
source /etc/birdnet/birdnet.conf
srv="birdnet_recording"
analyzing_now="."
counter=10
while [ 1 ];do
     sleep 61
     if [ $counter -le 0 ];then
        latest=$(cat ~/BirdNET-Pi/analyzing_now.txt)
        if [ "$latest" = "$analyzing_now" ];then
           echo "$(date) WARNING no change in analyzing_now for 10 iterations, restarting services"
           ~/BirdNET-Pi/scripts/restart_services.sh
        fi
        counter=10
        analyzing_now=$(cat ~/BirdNET-Pi/analyzing_now.txt)
     fi
     if [ -z "${RTSP_STREAM}" ];then
        ingest_dir=${RECS_DIR}/$(date +"%B-%Y/%d-%A")
        mkdir -p $ingest_dir
     else
        ingest_dir=$RECS_DIR/StreamData
     fi
     wavs=$(ls ${ingest_dir}/*.wav | wc -l)
     if [ $(systemctl --state=active | grep $srv | wc -l) -eq 0 ];then
        state="inactive"
     else
        state="active"
     fi
     echo "$(date)    INFO ${wavs} wav files waiting in ${ingest_dir}, $srv state is $state"
     if [ $wavs -gt 100 -a $state = "active" ];then
        sudo systemctl stop $srv
        echo "$(date) WARNING stopped $srv service"
     elif [ $wavs -le 100 -a $state = "inactive" ];then
        sudo systemctl start $srv
        echo "$(date)    INFO started $srv service"
     fi
     ((counter-=1))
done

Today I went out to the failing installation and found the following:

Spectrogram working.
Wav files were created.
Analysis had stopped and the log showed the following:

Seven days ago the analysis stopped after some old directories were removed. After "Restart Core Services" it all started again with analyzing all old files. I deleted the wav files and directories for all days except for today in order to see that everything worked in real time. Everything looked fine and reports are sent to BirdWeather again. https://app.birdweather.com/stations/236

I also found a message in the log that a subdirectory was missing in BirdSongs. Unfortunately I don't remember the name of it, but after adding the directory manually the message disappeared. I don't have that directory in my installations at home.

To be on the safe side (I hope) for that unmanned local installation I have now scheduled restart of services in crontab every six hours and also a reboot every night.

I too encountered issue with recordings and analyses stopped. I had been up since June 22, and then yesterday, after running an Tools > System Controls > Update the capture and analysis stopped happening,

I actually only noticed this today, and re-installed a couple times (using scripts/install_birdnet.sh) to no effect. The View Log was blank/not updating.

I moved/backed up directories out of way, and using the curl command I re-installed a fresh new install.

Nothing seemed to have any effect (still wasn't working). Then I saw and read this thread,

After Tools > Services > Restart core services analysis is functioning again.

I've had this happen, usually after I've been changing branches or updating. A restart of all services fixes things without fail.

Today it was happening very frequently. I would see "Stopping BirdNET Analysis" as the last thing in journalctl --no-pager.

It turns out my problem is a hardware one. My microphone finally succomed to a rainstorm we had, and while it still works most of the time, it causes the birdnet_recording service to have some kind of fault, and subsequently all services are told to stop.

Jul 06 10:51:02 birdnetpi systemd[1]: Stopping BirdNET Recording...
Jul 06 10:51:02 birdnetpi birdnet_recording.sh[1776244]: Aborted by signal Terminated...
Jul 06 10:51:02 birdnetpi sudo[1778452]:       pi : PWD=/home/pi/BirdSongs/Processed ; USER=root ; COMMAND=/usr/bin/systemctl stop birdnet_recording.service
Jul 06 10:51:02 birdnetpi sudo[1778452]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Jul 06 10:51:02 birdnetpi systemd[1]: birdnet_recording.service: Succeeded.
Jul 06 10:51:02 birdnetpi systemd[1]: Stopped BirdNET Recording.
Jul 06 10:51:02 birdnetpi systemd[1]: birdnet_recording.service: Consumed 2.139s CPU time.
Jul 06 10:51:02 birdnetpi sudo[1778449]: pam_unix(sudo:session): session closed for user root
Jul 06 10:51:02 birdnetpi sudo[1778452]: pam_unix(sudo:session): session closed for user root
Jul 06 10:51:02 birdnetpi sudo[1778454]:       pi : PWD=/home/pi/BirdSongs/Processed ; USER=root ; COMMAND=/usr/bin/systemctl stop custom_recording.service
Jul 06 10:51:02 birdnetpi sudo[1778454]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Jul 06 10:51:02 birdnetpi sudo[1778455]:       pi : PWD=/home/pi/BirdSongs/Processed ; USER=root ; COMMAND=/usr/bin/systemctl stop custom_recording.service
Jul 06 10:51:02 birdnetpi sudo[1778455]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Jul 06 10:51:02 birdnetpi sudo[1778455]: pam_unix(sudo:session): session closed for user root
Jul 06 10:51:02 birdnetpi sudo[1778454]: pam_unix(sudo:session): session closed for user root
Jul 06 10:51:02 birdnetpi sudo[1778459]:       pi : PWD=/home/pi/BirdSongs/Processed ; USER=root ; COMMAND=/usr/bin/systemctl stop birdnet_analysis.service
Jul 06 10:51:02 birdnetpi sudo[1778459]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Jul 06 10:51:02 birdnetpi sudo[1778458]:       pi : PWD=/home/pi/BirdSongs/Processed ; USER=root ; COMMAND=/usr/bin/systemctl stop birdnet_analysis.service
Jul 06 10:51:02 birdnetpi sudo[1778458]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Jul 06 10:51:02 birdnetpi systemd[1]: Stopping BirdNET Analysis...
Jul 06 10:51:02 birdnetpi systemd[1]: birdnet_analysis.service: Succeeded.
Jul 06 10:51:02 birdnetpi systemd[1]: Stopped BirdNET Analysis.
Jul 06 10:51:02 birdnetpi systemd[1]: birdnet_analysis.service: Consumed 16.610s CPU time.

Maybe that can provide some clues to anyone else having this issue.

I introduced a bug this morning that I squashed this afternoon -- it would have stopped recording for anyone every 3 minutes -- sorry, gang

Ahh, it was you and not my mic?!

Hi I have this same issue daily. The birdnet_server.service stops processing new audio files, they build up and system gets hours behind. Only option was to reboot as not long after it appears drops the wifi connection (PiOW2). I have enabled remote log output so was able to capture some errors:

2022-07-23T11:19:28+00:00 birdnetpi1 python3[79649]: Date-time: 2022-07-23 10:13:33 2022-07-23T11:19:47+00:00 birdnetpi1 python3[79649]: ANALYZING AUDIO... Exception in thread Thread-569: 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: Traceback (most recent call last): 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: self.run() 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: File "/usr/lib/python3.9/threading.py", line 892, in run 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: self._target(*self._args, **self._kwargs) 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: File "/usr/local/bin/server.py", line 408, in handle_client 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: sendAppriseNotifications(species, 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: File "/home/pi/BirdNET-Pi/scripts/utils/notifications.py", line 95, in sendAppriseNotifications 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: notify(notify_body, notify_title, image_url) 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: File "/home/pi/BirdNET-Pi/scripts/utils/notifications.py", line 25, in notify 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: apobj.notify( 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: File "/home/pi/BirdNET-Pi/birdnet/lib/python3.9/site-packages/apprise/Apprise.py", line 374, in notify 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: return py3compat.asyncio.tosync( 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: File "/home/pi/BirdNET-Pi/birdnet/lib/python3.9/site-packages/apprise/py3compat/asyncio.py", line 66, in tosync 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: return asyncio.run(cor, debug=debug) 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: File "/usr/lib/python3.9/asyncio/runners.py", line 49, in run 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: loop.run_until_complete(loop.shutdown_default_executor()) 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete 2022-07-23T11:19:48+00:00 birdnetpi1 python3[79649]: return future.result() 2022-07-23T11:19:49+00:00 birdnetpi1 python3[79649]: File "/usr/lib/python3.9/asyncio/base_events.py", line 563, in shutdown_default_executor 2022-07-23T11:19:49+00:00 birdnetpi1 python3[79649]: thread.start() 2022-07-23T11:19:49+00:00 birdnetpi1 python3[79649]: File "/usr/lib/python3.9/threading.py", line 874, in start 2022-07-23T11:19:49+00:00 birdnetpi1 python3[79649]: _start_new_thread(self._bootstrap, ()) 2022-07-23T11:19:49+00:00 birdnetpi1 python3[79649]: RuntimeError: can't start new thread 2022-07-23T11:38:56+00:00 birdnetpi1 python3[136833]: INFO: Created TensorFlow Lite XNNPACK delegate for CPU. 2022-07-23T11:38:56+00:00 birdnetpi1 python3[136833]: INFO: Created TensorFlow Lite delegate for select TF ops.

Key seems to be RuntimeError: can't start new thread With this occurrence I had terminal access:

birdnetpi1:~/BirdNET-Pi$ sudo systemctl status birdnet_server.service ● birdnet_server.service - BirdNET Analysis Server Loaded: loaded (/home/pi/BirdNET-Pi/templates/birdnet_server.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2022-07-23 09:03:05 AEST; 2h 34min ago Main PID: 79649 (python3) Tasks: 189 (limit: 191) CPU: 1h 6min 32.916s CGroup: /system.slice/birdnet_server.service └─79649 /home/pi/BirdNET-Pi/birdnet/bin/python3 /usr/local/bin/server.py

Jul 23 11:19:48 birdnetpi1 python3[79649]: return asyncio.run(cor, debug=debug) Jul 23 11:19:48 birdnetpi1 python3[79649]: File "/usr/lib/python3.9/asyncio/runners.py", line 49, in run Jul 23 11:19:48 birdnetpi1 python3[79649]: loop.run_until_complete(loop.shutdown_default_executor()) Jul 23 11:19:48 birdnetpi1 python3[79649]: File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete Jul 23 11:19:48 birdnetpi1 python3[79649]: return future.result() Jul 23 11:19:48 birdnetpi1 python3[79649]: File "/usr/lib/python3.9/asyncio/base_events.py", line 563, in shutdown_default_executor Jul 23 11:19:48 birdnetpi1 python3[79649]: thread.start() Jul 23 11:19:48 birdnetpi1 python3[79649]: File "/usr/lib/python3.9/threading.py", line 874, in start Jul 23 11:19:48 birdnetpi1 python3[79649]: _start_new_thread(self._bootstrap, ()) Jul 23 11:19:48 birdnetpi1 python3[79649]: RuntimeError: can't start new thread

Here is another one from today however I could no longer access the device as it would not accept the password copy and pasted from keepass :|

2022-07-24T07:56:50+00:00 birdnetpi1 python3[329]: ANALYZING AUDIO... DONE! Time 9.5 SECONDS 2022-07-24T07:56:50+00:00 birdnetpi1 python3[329]: WRITING RESULTS TO /home/pi/BirdSongs/July-2022/24-Sunday/2022-07-24-birdnet-07:41:00.wav.csv ... DONE! WROTE 1 RESULTS. 2022-07-24T07:56:50+00:00 birdnetpi1 python3[329]: 2022-07-24;07:41:00;Gymnorhina tibicen;Australian Magpie;0.76111114;-35.2267;149.1171;0.7;29;1.25;0.0;Australian_Magpie-76-2022-07-24-birdnet-07:41:00.mp3 2022-07-24T07:56:50+00:00 birdnetpi1 python3[329]: Soundscape POST Response Status - 201 2022-07-24T07:56:50+00:00 birdnetpi1 python3[329]: { "timestamp": "2022-07-24T07:41:27+10:00","lat": -35.2267,"lon": 149.1171,"soundscapeId": 16085304,"soundscapeStartTime": 27.0,"soundscapeEndTime": 30.0,"commonName": "Australian Magpie","scientificName": "Gymnorhina tibicen","algorithm": "alpha","confidence": 0.76111114 } 2022-07-24T07:56:50+00:00 birdnetpi1 python3[329]: Detection POST Response Status - 201 2022-07-24T07:56:52+00:00 birdnetpi1 python3[329]: READING AUDIO DATA... DONE! READ 10 CHUNKS. 2022-07-24T07:56:52+00:00 birdnetpi1 python3[329]: Date-time: 2022-07-24 07:41:30 2022-07-24T07:57:02+00:00 birdnetpi1 CRON[458783]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T07:57:02+00:00 birdnetpi1 CRON[458784]: (pi) CMD (/usr/local/bin/cleanup.sh >/dev/null 2>&1) 2022-07-24T07:57:02+00:00 birdnetpi1 CRON[458783]: pam_unix(cron:session): session closed for user pi 2022-07-24T07:57:59+00:00 birdnetpi1 kernel: [50594.674046] cgroup: fork rejected by pids controller in /system.slice/birdnet_server.service 2022-07-24T07:57:59+00:00 birdnetpi1 python3[329]: ANALYZING AUDIO... Exception in thread Thread-2178: 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: Traceback (most recent call last): 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: self.run() 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: File "/usr/lib/python3.9/threading.py", line 892, in run 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: self._target(*self._args, **self._kwargs) 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: File "/usr/local/bin/server.py", line 408, in handle_client 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: sendAppriseNotifications(species, 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: File "/home/pi/BirdNET-Pi/scripts/utils/notifications.py", line 95, in sendAppriseNotifications 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: notify(notify_body, notify_title, image_url) 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: File "/home/pi/BirdNET-Pi/scripts/utils/notifications.py", line 25, in notify 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: apobj.notify( 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: File "/home/pi/BirdNET-Pi/birdnet/lib/python3.9/site-packages/apprise/Apprise.py", line 374, in notify 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: return py3compat.asyncio.tosync( 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: File "/home/pi/BirdNET-Pi/birdnet/lib/python3.9/site-packages/apprise/py3compat/asyncio.py", line 66, in tosync 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: return asyncio.run(cor, debug=debug) 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: File "/usr/lib/python3.9/asyncio/runners.py", line 49, in run 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: loop.run_until_complete(loop.shutdown_default_executor()) 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: return future.result() 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: File "/usr/lib/python3.9/asyncio/base_events.py", line 563, in shutdown_default_executor 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: thread.start() 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: File "/usr/lib/python3.9/threading.py", line 874, in start 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: _start_new_thread(self._bootstrap, ()) 2022-07-24T07:58:00+00:00 birdnetpi1 python3[329]: RuntimeError: can't start new thread

after this occurs the log is pretty much just:

2022-07-24T08:00:01+00:00 birdnetpi1 CRON[459862]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T08:00:02+00:00 birdnetpi1 CRON[459863]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T08:00:02+00:00 birdnetpi1 CRON[459869]: (pi) CMD (/usr/local/bin/cleanup.sh >/dev/null 2>&1) 2022-07-24T08:00:03+00:00 birdnetpi1 CRON[459870]: (pi) CMD (/usr/local/bin/disk_check.sh >/dev/null 2>&1) 2022-07-24T08:00:03+00:00 birdnetpi1 CRON[459862]: pam_unix(cron:session): session closed for user pi 2022-07-24T08:00:03+00:00 birdnetpi1 CRON[459863]: pam_unix(cron:session): session closed for user pi 2022-07-24T08:03:01+00:00 birdnetpi1 CRON[460990]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T08:03:01+00:00 birdnetpi1 CRON[460991]: (pi) CMD (/usr/local/bin/cleanup.sh >/dev/null 2>&1) 2022-07-24T08:03:01+00:00 birdnetpi1 CRON[460990]: pam_unix(cron:session): session closed for user pi 2022-07-24T08:05:02+00:00 birdnetpi1 CRON[461780]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T08:05:02+00:00 birdnetpi1 CRON[461781]: (pi) CMD (/usr/local/bin/disk_check.sh >/dev/null 2>&1) 2022-07-24T08:05:03+00:00 birdnetpi1 CRON[461780]: pam_unix(cron:session): session closed for user pi 2022-07-24T08:06:01+00:00 birdnetpi1 CRON[462147]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T08:06:01+00:00 birdnetpi1 CRON[462148]: (pi) CMD (/usr/local/bin/cleanup.sh >/dev/null 2>&1) 2022-07-24T08:06:01+00:00 birdnetpi1 CRON[462147]: pam_unix(cron:session): session closed for user pi 2022-07-24T08:09:01+00:00 birdnetpi1 CRON[463328]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0) 2022-07-24T08:09:01+00:00 birdnetpi1 CRON[463327]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T08:09:01+00:00 birdnetpi1 CRON[463329]: (pi) CMD (/usr/local/bin/cleanup.sh >/dev/null 2>&1) 2022-07-24T08:09:01+00:00 birdnetpi1 CRON[463330]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi) 2022-07-24T08:09:01+00:00 birdnetpi1 CRON[463327]: pam_unix(cron:session): session closed for user pi 2022-07-24T08:09:01+00:00 birdnetpi1 CRON[463328]: pam_unix(cron:session): session closed for user root 2022-07-24T08:09:02+00:00 birdnetpi1 systemd[1]: Starting Clean php session files... 2022-07-24T08:09:12+00:00 birdnetpi1 systemd[1]: phpsessionclean.service: Succeeded. 2022-07-24T08:09:12+00:00 birdnetpi1 systemd[1]: Finished Clean php session files.

The Pi0W2 is severely RAM limited. Is it not cleaning up after itself? Tasks: 189 (limit: 191)

I am happy to assist if possible. I will look at https://github.com/JohnButcher script :+1:

edit: I doubt the suse link is relevant. I am going to try monitor the syslog for reoccuring error messages and try an action from that.

A bit more information from another occurrence. I can see errors, I just don't know what they mean :)

The entry I have bolded below may have a solution found here

2022-07-24T10:41:12+00:00 birdnetpi1 birdnet_analysis.sh[27933]: RECORDING_LENGTH set to 30 2022-07-24T10:41:12+00:00 birdnetpi1 birdnet_analysis.sh[27933]: /home/pi/BirdNET-Pi/birdnet/bin/python3 /home/pi/BirdNET-Pi/scripts/analyze.py --i /home/pi/BirdSongs/July-2022/24-Sunday/2022-07-24-birdnet-09:27:22.wav --o /home/pi/BirdSongs/July-2022/24-Sunday/2022-07-24-birdnet-09:27:22.wav.csv --lat -34.XX --lon 149.XX --week 28 --overlap 0.0 --sensitivity 1.25 --min_conf 0.7 --exclude_list /home/pi/BirdNET-Pi/exclude_species_list.txt --birdweather_id "IN_USE" 2022-07-24T10:41:12+00:00 birdnetpi1 python3[334]: ANALYZING AUDIO... DONE! Time 9.6 SECONDS 2022-07-24T10:41:12+00:00 birdnetpi1 python3[334]: WRITING RESULTS TO /home/pi/BirdSongs/July-2022/24-Sunday/2022-07-24-birdnet-09:26:52.wav.csv ... DONE! WROTE 0 RESULTS. 2022-07-24T10:41:15+00:00 birdnetpi1 python3[334]: READING AUDIO DATA... DONE! READ 10 CHUNKS. 2022-07-24T10:41:15+00:00 birdnetpi1 python3[334]: Date-time: 2022-07-24 09:27:22 2022-07-24T10:41:46+00:00 birdnetpi1 kernel: [ 6449.052673] cgroup: fork rejected by pids controller in /system.slice/birdnet_server.service 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: ANALYZING AUDIO... Exception in thread Thread-565: 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: Traceback (most recent call last): 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: self.run() 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: File "/usr/lib/python3.9/threading.py", line 892, in run 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: self._target(*self._args, **self._kwargs) 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: File "/usr/local/bin/server.py", line 408, in handle_client 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: sendAppriseNotifications(species, 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: File "/home/pi/BirdNET-Pi/scripts/utils/notifications.py", line 95, in sendAppriseNotifications 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: notify(notify_body, notify_title, image_url) 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: File "/home/pi/BirdNET-Pi/scripts/utils/notifications.py", line 25, in notify 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: apobj.notify( 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: File "/home/pi/BirdNET-Pi/birdnet/lib/python3.9/site-packages/apprise/Apprise.py", line 374, in notify 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: return py3compat.asyncio.tosync( 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: File "/home/pi/BirdNET-Pi/birdnet/lib/python3.9/site-packages/apprise/py3compat/asyncio.py", line 66, in tosync 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: return asyncio.run(cor, debug=debug) 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: File "/usr/lib/python3.9/asyncio/runners.py", line 49, in run 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: loop.run_until_complete(loop.shutdown_default_executor()) 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: return future.result() 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: File "/usr/lib/python3.9/asyncio/base_events.py", line 563, in shutdown_default_executor 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: thread.start() 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: File "/usr/lib/python3.9/threading.py", line 874, in start 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: _start_new_thread(self._bootstrap, ()) 2022-07-24T10:41:46+00:00 birdnetpi1 python3[334]: RuntimeError: can't start new thread 2022-07-24T10:42:01+00:00 birdnetpi1 CRON[44855]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T10:42:01+00:00 birdnetpi1 CRON[44856]: (pi) CMD (/usr/local/bin/cleanup.sh >/dev/null 2>&1) 2022-07-24T10:42:01+00:00 birdnetpi1 CRON[44855]: pam_unix(cron:session): session closed for user pi 2022-07-24T10:45:01+00:00 birdnetpi1 CRON[46017]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T10:45:04+00:00 birdnetpi1 CRON[46016]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T10:45:04+00:00 birdnetpi1 CRON[46018]: (pi) CMD (/usr/local/bin/cleanup.sh >/dev/null 2>&1) 2022-07-24T10:45:04+00:00 birdnetpi1 CRON[46019]: (pi) CMD (/usr/local/bin/disk_check.sh >/dev/null 2>&1) 2022-07-24T10:45:04+00:00 birdnetpi1 CRON[46016]: pam_unix(cron:session): session closed for user pi 2022-07-24T10:45:05+00:00 birdnetpi1 CRON[46017]: pam_unix(cron:session): session closed for user pi 2022-07-24T10:48:01+00:00 birdnetpi1 CRON[47109]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T10:48:01+00:00 birdnetpi1 CRON[47110]: (pi) CMD (/usr/local/bin/cleanup.sh >/dev/null 2>&1) 2022-07-24T10:48:01+00:00 birdnetpi1 CRON[47109]: pam_unix(cron:session): session closed for user pi 2022-07-24T10:50:01+00:00 birdnetpi1 CRON[47856]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T10:50:01+00:00 birdnetpi1 CRON[47857]: (pi) CMD (/usr/local/bin/disk_check.sh >/dev/null 2>&1) 2022-07-24T10:50:02+00:00 birdnetpi1 CRON[47856]: pam_unix(cron:session): session closed for user pi 2022-07-24T10:51:01+00:00 birdnetpi1 CRON[48246]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T10:51:01+00:00 birdnetpi1 CRON[48247]: (pi) CMD (/usr/local/bin/cleanup.sh >/dev/null 2>&1) 2022-07-24T10:51:01+00:00 birdnetpi1 CRON[48246]: pam_unix(cron:session): session closed for user pi 2022-07-24T10:54:01+00:00 birdnetpi1 CRON[49425]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T10:54:01+00:00 birdnetpi1 CRON[49426]: (pi) CMD (/usr/local/bin/cleanup.sh >/dev/null 2>&1) 2022-07-24T10:54:01+00:00 birdnetpi1 CRON[49425]: pam_unix(cron:session): session closed for user pi 2022-07-24T10:54:31+00:00 birdnetpi1 rngd[434]: stats: bits received from HRNG source: 120064 2022-07-24T10:54:31+00:00 birdnetpi1 rngd[434]: stats: bits sent to kernel pool: 62560 2022-07-24T10:54:31+00:00 birdnetpi1 rngd[434]: stats: entropy added to kernel pool: 62560 2022-07-24T10:54:31+00:00 birdnetpi1 rngd[434]: stats: FIPS 140-2 successes: 6 2022-07-24T10:54:31+00:00 birdnetpi1 rngd[434]: stats: FIPS 140-2 failures: 0 2022-07-24T10:54:31+00:00 birdnetpi1 rngd[434]: stats: FIPS 140-2(2001-10-10) Monobit: 0 2022-07-24T10:54:31+00:00 birdnetpi1 rngd[434]: stats: FIPS 140-2(2001-10-10) Poker: 0 2022-07-24T10:54:31+00:00 birdnetpi1 rngd[434]: stats: FIPS 140-2(2001-10-10) Runs: 0 2022-07-24T10:54:31+00:00 birdnetpi1 rngd[434]: stats: FIPS 140-2(2001-10-10) Long run: 0 2022-07-24T10:54:31+00:00 birdnetpi1 rngd[434]: stats: FIPS 140-2(2001-10-10) Continuous run: 0 2022-07-24T10:54:31+00:00 birdnetpi1 rngd[434]: stats: HRNG source speed: (min=759.587; avg=801.902; max=819.917)Kibits/s 2022-07-24T10:54:31+00:00 birdnetpi1 rngd[434]: stats: FIPS tests speed: (min=9.447; avg=20.876; max=43.251)Mibits/s 2022-07-24T10:54:31+00:00 birdnetpi1 rngd[434]: stats: Lowest ready-buffers level: 2 2022-07-24T10:54:31+00:00 birdnetpi1 rngd[434]: stats: Entropy starvations: 0 2022-07-24T10:54:31+00:00 birdnetpi1 rngd[434]: stats: Time spent starving for entropy: (min=0; avg=0.000; max=0)us 2022-07-24T10:55:01+00:00 birdnetpi1 CRON[49816]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T10:55:01+00:00 birdnetpi1 CRON[49817]: (pi) CMD (/usr/local/bin/disk_check.sh >/dev/null 2>&1) 2022-07-24T10:55:02+00:00 birdnetpi1 CRON[49816]: pam_unix(cron:session): session closed for user pi 2022-07-24T10:57:01+00:00 birdnetpi1 CRON[50596]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T10:57:01+00:00 birdnetpi1 CRON[50597]: (pi) CMD (/usr/local/bin/cleanup.sh >/dev/null 2>&1) 2022-07-24T10:57:01+00:00 birdnetpi1 CRON[50596]: pam_unix(cron:session): session closed for user pi 2022-07-24T11:00:02+00:00 birdnetpi1 CRON[51763]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T11:00:02+00:00 birdnetpi1 CRON[51762]: pam_unix(cron:session): session opened for user pi(uid=1000) by (uid=0) 2022-07-24T11:00:02+00:00 birdnetpi1 CRON[51764]: (pi) CMD (/usr/local/bin/cleanup.sh >/dev/null 2>&1) 2022-07-24T11:00:02+00:00 birdnetpi1 CRON[51765]: (pi) CMD (/usr/local/bin/disk_check.sh >/dev/null 2>&1) 2022-07-24T11:00:02+00:00 birdnetpi1 CRON[51762]: pam_unix(cron:session): session closed for user pi 2022-07-24T11:00:03+00:00 birdnetpi1 CRON[51763]: pam_unix(cron:session): session closed for user pi

mcguirepr89 / BirdNET-Pi

Action when analyzes are stopped #393