search

mcguirepr89 / BirdNET-Pi

A realtime acoustic bird classification system for the Raspberry Pi 4B, 3B+, and 0W2 built on the TFLite version of BirdNET.
Other
1.28k stars 136 forks source link

birdnet gets stuck on analysing old recording #1078

Open mnhn-paul opened 9 months ago

mnhn-paul commented 9 months ago

Describe the bug

My BirdNetPi installation on a Proxmox VM was now running fine some a week, but today it simply stopped detecting new occurrences. I check the logs and nothing is really happening over there .

19:46:22---BirdNET-Pi/birdnet/bin/python3 BirdNET-Pi/scripts/analyze.py --i BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-19:10:40.wav --o BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-19:10:40.wav.csv --lat 50.XX --lon 6.XX --week 44 --overlap 1 --sensitivity 1.25 --min_conf 0.70 --exclude_list BirdNET-Pi/exclude_species_list.txt --birdweather_id "IN_USE"
19:46:22---ANALYZING AUDIO... DONE! Time 0.3 SECONDS
19:46:22---WRITING RESULTS TO BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-19:10:19.wav.csv ... DONE! WROTE 0 RESULTS.
19:46:22---READING AUDIO DATA... DONE! READ 0 CHUNKS.
19:46:22---Date-time: 2023-11-26 19:10:40

I did try the usual thing, reboot, stopping/restarting services but nothing seems to really help. It simply seems to be stuck at an old recoding but nothing really happens.

image

Live audio stream also seems to work fine.

Restarting birdnet_server.service seem to sort of help, as afterwards a new file was being analysed, However only one before stopping again.

19:46:22---BirdNET-Pi/birdnet/bin/python3 BirdNET-Pi/scripts/analyze.py --i BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-19:10:40.wav --o BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-19:10:40.wav.csv --lat 50.XX --lon 6.XX --week 44 --overlap 1 --sensitivity 1.25 --min_conf 0.70 --exclude_list BirdNET-Pi/exclude_species_list.txt --birdweather_id "IN_USE"
19:46:22---ANALYZING AUDIO... DONE! Time 0.3 SECONDS
19:46:22---WRITING RESULTS TO BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-19:10:19.wav.csv ... DONE! WROTE 0 RESULTS.
19:46:22---READING AUDIO DATA... DONE! READ 0 CHUNKS.
19:46:22---Date-time: 2023-11-26 19:10:40
19:53:57---Files loaded
19:53:58---RECORDING_LENGTH set to 15
19:53:58---BirdNET-Pi/birdnet/bin/python3 BirdNET-Pi/scripts/analyze.py --i BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-19:10:47.wav --o BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-19:10:47.wav.csv --lat 50.XX --lon 6.XX --week 44 --overlap 1 --sensitivity 1.25 --min_conf 0.70 --exclude_list BirdNET-Pi/exclude_species_list.txt --birdweather_id "IN_USE"
19:53:58---ANALYZING AUDIO... DONE! Time 0.0 SECONDS
19:53:58---WRITING RESULTS TO BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-19:10:40.wav.csv ... DONE! WROTE 0 RESULTS.
19:53:58---READING AUDIO DATA... DONE! READ 0 CHUNKS.
19:53:58---Date-time: 2023-11-26 19:10:47

I restarted the birdnet_server.service here:

19:54:14---ANALYZING AUDIO...
19:54:15---INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
19:54:16---Files loaded
19:54:16---RECORDING_LENGTH set to 15
19:54:16---BirdNET-Pi/birdnet/bin/python3 BirdNET-Pi/scripts/analyze.py --i BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-19:10:55.wav --o BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-19:10:55.wav.csv --lat 50.XX --lon 6.XX --week 44 --overlap 1 --sensitivity 1.25 --min_conf 0.70 --exclude_list BirdNET-Pi/exclude_species_list.txt --birdweather_id "IN_USE"
19:54:16---LOADING TF LITE MODEL... DONE!
19:54:17---READING AUDIO DATA... DONE! READ 8 CHUNKS.
19:54:17---Date-time: 2023-11-26 19:10:55
19:54:18---0.0;3.0-('Phalaenoptilus nuttallii_Common Poorwill', 0.101125255)
19:54:18---2.0;5.0-('Spizella pallida_Clay-colored Sparrow', 0.10355277)
19:54:18---4.0;7.0-('Phalaenoptilus nuttallii_Common Poorwill', 0.19761342)
19:54:18---6.0;9.0-("Setophaga kirtlandii_Kirtland's Warbler", 0.13450275)
19:54:18---8.0;11.0-('Phalaenoptilus nuttallii_Common Poorwill', 0.12653181)
19:54:18---10.0;13.0-('Phalaenoptilus nuttallii_Common Poorwill', 0.09327367)
19:54:18---12.0;15.0-('Phalaenoptilus nuttallii_Common Poorwill', 0.069007814)
19:54:18---14.0;17.0-('Spizella pallida_Clay-colored Sparrow', 0.07210439)
19:54:18---RECORDING_LENGTH set to 15
19:54:18---BirdNET-Pi/birdnet/bin/python3 BirdNET-Pi/scripts/analyze.py --i BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-19:11:17.wav --o BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-19:11:17.wav.csv --lat 50.XX --lon 6.XX --week 44 --overlap 1 --sensitivity 1.25 --min_conf 0.70 --exclude_list BirdNET-Pi/exclude_species_list.txt --birdweather_id "IN_USE"
19:54:18---ANALYZING AUDIO... loaded META model
19:54:18---DONE! Time 0.4 SECONDS
19:54:18---WRITING RESULTS TO BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-19:10:55.wav.csv ... DONE! WROTE 0 RESULTS.
19:54:18---READING AUDIO DATA... DONE! READ 0 CHUNKS.
19:54:18---Date-time: 2023-11-26 19:11:17

And Then nothing happens again...

I noticed that the BirdSongs/StreamData folder is filling up with files that are not being processed.

image

So I simply deleted all the files in the folder to see what happens, I thought maybe the files are corrupted or there is a backlog. After deleting I restarted all core services, this seems to help and the logs show what is to be expected:

20:00:19---DONE! Time 0.2 SECONDS
20:00:19---WRITING RESULTS TO BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-19:58:52.wav.csv ... DONE! WROTE 0 RESULTS.
20:00:19---READING AUDIO DATA... DONE! READ 3 CHUNKS.
20:00:19---Date-time: 2023-11-26 19:59:38
20:00:19---0.0;3.0-('Human_Human', 0.0)
20:00:19---2.0;5.0-('Human_Human', 0.0)
20:00:19---4.0;7.0-('Athene brama_Spotted Owlet', 0.0826815)
20:00:29---RECORDING_LENGTH set to 15
20:00:35---BirdNET-Pi/birdnet/bin/python3 BirdNET-Pi/scripts/analyze.py --i BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-20:00:17.wav --o BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-20:00:17.wav.csv --lat 50.XX --lon 6.XX --week 44 --overlap 1 --sensitivity 1.25 --min_conf 0.70 --exclude_list BirdNET-Pi/exclude_species_list.txt --birdweather_id "IN_USE"
20:00:35---ANALYZING AUDIO... DONE! Time 0.1 SECONDS
20:00:35---WRITING RESULTS TO BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-19:59:38.wav.csv ... DONE! WROTE 0 RESULTS.

but only for a short time, as only a few minutes later its getting stuck again:

20:04:04---RECORDING_LENGTH set to 15
20:04:06---BirdNET-Pi/birdnet/bin/python3 BirdNET-Pi/scripts/analyze.py --i BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-20:03:14.wav --o BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-20:03:14.wav.csv --lat 50.XX --lon 6.XX --week 44 --overlap 1 --sensitivity 1.25 --min_conf 0.70 --exclude_list BirdNET-Pi/exclude_species_list.txt --birdweather_id "IN_USE"
20:04:06---ANALYZING AUDIO... DONE! Time 0.2 SECONDS
20:04:06---WRITING RESULTS TO BirdSongs/StreamData/2023-11-26-birdnet-RTSP_1-20:03:00.wav.csv ... DONE! WROTE 0 RESULTS.
20:04:06---READING AUDIO DATA... DONE! READ 0 CHUNKS.
20:04:06---Date-time: 2023-11-26 20:03:14

My build I'm running birdnet on a proxmox VM following this tutorial. The audio comes from a security camera stream.

image

hope someone can help out

mnhn-paul commented 9 months ago

Yesterday evening I set the overlap to "0" and deleted the contents of StreamData and restarted everything, it did work again, but only till early this morning...

also gave the VM (a LSC container actually) more resources (cores and ram) but that does not seem to help.

mnhn-paul commented 9 months ago

Did some more investigations, it seem that the issues comes from recordings which are being created and are to short (less then 3 seconds). This seems to be coming from connection issues with the streaming camera I'm using. here are some of the files which are created and then are not being processed correctly. StreamData_231129_054833.zip

Ralphy140 commented 7 months ago

did you ever get to the bottom of this? I am having the exact same issue (well same symptoms) I just removed the streamdata folder so cant check for the short recordings. Out of curiosity what camera are you using for the mic? I have Reolink cameras setup and i am using the sub RTSP stream to pickup the audio

mnhn-paul commented 7 months ago

@Ralphy140 unfortunately I did not solve the issue, ..

in the end I installed "official" birdnetpi on an Pi4 and migrated the data, since then I only got the issue about 2 or 3 times. At that point there was a series of very short or corrupted audio files which were analyzed, I think one takes about 15min before moving to the next, if there are hundreds this will basically block birdnet and recordings are accumulating.

When encountering the issue, I connected to the pi and deleted all the extracted files to be analyzed below 2MB or so, then I restarted the services and the backlog was being processed.

I'm also using a Reolink 510WA over WIFI, maybe WIFI connection issues are at the origin of the bad recordings?

Ralphy140 commented 7 months ago

Ahh bugger, Can i ask what you mean by "Offical" ? And also interesting on the reolink camera, I am using Reolink cameras but the 822 and the are wired in. I wonder if it has something to do with the way they process the RTSP stream

mnhn-paul commented 7 months ago

With official I meed the one in this repo, in opposite the the one from here https://github.com/MatthewBCooke/BirdNET-Pi which I used in my initial install on a VM, I'm not to sure in how may they actually differ.

Concerning the camera I was sort of able to reproduce the issue by turning off or rebooting the camera, birdnetpi probably then does not relay know how to handle the lack of stream and creates these file that it is not able to analyze.

PeterMescher commented 6 months ago

I had a similar problem (sometimes my files from my RTSP stream are too long, and birdnet chokes.) To work-around the issue, I had chatgpt write a script that simply deleted the oldest file in the StreamData-Analyzed dir if it was older than 30 seconds. (Birdnet only takes 0.3 seconds to analyze my sounds.) This has kept the problem from re-occurring, at the cost of losing a handful of recordings (generally no more than 2-3 on a bad day.)

csv_cleanup.txt

(Save this file with a .sh extension in your /root and chmod 755 it to make it executable)

Then I put a line in my crontab to run it once every 5 minutes, and redirect the output and stderr to null: (If you don't, you get an error message about there being no files for stat to evaluate.)

5 * /root/csv_cleanup.sh > /dev/null 2>&1 &

Ralphy140 commented 5 months ago

I seemed to have resolved this issue by running the RTSP from the Camera through go2rtc (https://github.com/AlexxIT/go2rtc) this lets you ingest the HTTP stream from the camera and then output a clean RTSP stream. I think the issue is caused by the way the Reolink Camera generates the RTSP stream combined with poor error handling in the BirdNet backend. I suspect that if you do both as above and this you will be golden. Ideally the birdnet error handling would be uplifted but this is a good enough workaround for now.