Feature request: Removable USB Storage

[DD4WH]

I am looking for the audio files that have been recorded in the last two days, but I could not find them.

• /BirdSongs/Extracted/Processed does not contain any files
• /BirdSongs/Processed does only contain a collection of some files recorded during the day
• there seems to be some kind of bug in the calculation of which files are being deleted in the "ring buffer" ?
• But maybe I have misinterpreted that ALL the recordings of the last 48 hours are being preserved?

[mcguirepr89]

Disk space management now only keeps the original audio recordings if a detection was found therein. Then, of those files, only the most recent 100 are kept. There may be more or less than that at any time, but the cleanup cron goes through and culls any audio file for which the BirdNET analysis detected nothing above the minimum confidence level.

I'm really sorry if this does away with data you would have found useful. If it does, then you can edit the cron job to only fire every two days at 1AM by turning this:

#birdnet
*/3 * * * * /usr/local/bin/cleanup.sh &> /dev/n
#birdnet
@reboot /usr/local/bin/cleanup.sh &> /dev/null

into this:

#birdnet
0 1 */2 * * /usr/local/bin/cleanup.sh &> /dev/null

[DD4WH]

Excellent, thanks for this! I changed my crontab and now I can access all the recordings from today!

The idea behind is the following scenario:

• BirdNET classified a particularly interesting bird and extracted only one recording with one call
• I suspect there could be more calls of this particular species right before and after that recording which did not become classified with a confidence score higher than the preset threshold
• Then I access the PROCESSED folder and can find those recordings with more calls, but lower ID score
• If I do that every few days, I should not miss anything ;-)
• the only disadvantage of doing it like this is that you need a larger SD card
• Audio file size of 24 hours of recording is 48000 samples per second 2 bytes (16 bit) 60 seconds per minute 60 minutes per hour 24 hours == 8.3 Gigabytes audio size per day
• If you use a 64 Gigabyte micro-SD-card, you should be able to store at least 2 to 4 full days of audio

[DD4WH]

well, this naturally calls for another feature wish:

• to be able to plug-in a USB stick and have it automatically detected and used for storage of the extracted files (and also the temporarily stored processed files)
• it would be desirable to be able to unplug the USB stick (copy the files to a computer) and insert the empty stick again
• should I open a new issue/feature wish for this?

[mcguirepr89]

Sure! a new issue/feature wish would be great. I can keep you updated on progress. I have a lot of things I want to do for BirdNET-Pi and tend to oil the squeakiest wheel, so to speak :) so requests and reminders and follow ups go a long way! Otherwise I forget and move onto the next thing. And I forget to update my repo "projects," but I never forget about issues

[mcguirepr89]

Update

Wowee Zowee!! this is harder than I expected. Its a Good News and Bad News situation:

• The Good News: Configuring the BirdNET-Pi to handle a USB is not an issue. I have been able to set up an easy little button that you press after inserting the USB that will configure the stick and build the directory structure it needs to act as the storage for data. This can be done for any external storage device and it doesn't have to be the same device each time. You can stop services, remove the old, plug in the new, press the button to configure the new usb, and then restart services.

• The Bad News: Windows and Mac filesystems make this setup kind of tricky. The file system that works best across all operating systems and filesystem types is vFAT or FAT32. Data stored on a vFAT filesystem can plug and play in Linux, MacOS, and Windows without any trouble. The trouble is that the vFAT filesystem sucks, and cannot support being written to in a streaming format. This means you can plug the USB in and it will build the necessary directory structure just fine, but won't allow the recording service to write to it AND it doesn't support any linking, so all files have to be coppied over to the USB (html, php, and shell scripts) AND none of the extraction symlinking will work, so ONLY By_Date would even be possible IF the recording service could even write to the disk.

• A solution Windows and macOS client software to handle ext4 filesystem formats

Update:

• Another solution I could change my thinking on this and instead of having the external USB mount to the BirdSongs directory, it could mount into another directory and I can alter the extraction service to put files there after processing. This works around the issues I'm seeing. I think this is likely the route I will take, though I will have to figure out what to do about the Caddyfile. :thinking:

[mcguirepr89]

I'm going to rename this so that it can act as a USB feature request

[mcguirepr89]

I'm still working on this, but in the meantime, you can utilize SD cards with the pre-installed images and just swap them out and setup the new one (takes about 8-10 minutes total). Then take the full SD card home and take from it what you need. You will probably need some kind of Windows solution to mount the Linux filesystem (ext4), but I'm sure there is something out there for that.)

I'lll let you know when this feature is baked in

[DD4WH]

@mcguirepr89 Patrick, just a short note about an experience from today: I had my Birdnet-Pi in the field for 36 days running and recording perfectly and I was able to see what is happening by connecting via an LTE router. However, today I wanted to copy all the extractions from the SD card to a USB stick. I realized that two days ago, the SD card had run full and now the system was not able to boot any more and I was unable to get the recordings from the SD card, because obviously, the system had killed itself by overwriting important parts of the OS . . . It does not matter, because its late autumn/winter here and the birds are not that interesting at this time of the year. But it makes me think about a standardized way to get the recordings from the SD card in such a situation before the OS overwrites itself . . . [it was a 64GB SD card] I will try again by setting up a new 400GB SD card with your system. And I will take care to insert a USB stick every month and copy the data from the system as a backup and delete old data. Maybe you could implement a stop script that prevents overwriting of the system when the SD card is full of recordings? That would help prevent unpleasant surprises ;-).

A good solution for getting the data from the system would be to insert a USB stick and then click on a script button that would copy all the extractions to the stick and afterwards delete them from the SD card (but leave the data in the database untouched).

[mcguirepr89]

Frank, thank you so much for this insight. I wasn't aware that you had a BirdNET-Pi running for so long -- pretty cool.

I'm sorry, though, that the system filled up.

I was unable to get the recordings from the SD card, because obviously, the system had killed itself by overwriting important parts of the OS . . .

You should be able to insert the SD card into a computer (I don't know if Windows will be able to do this, but inserting it into another BirdNET-Pi would certainly work) and get to the BirdSongs directory that way. LMK if you'd like a hand with getting that data -- I know that it is still there (it would not have been overwritten, merely insufficient storage to boot properly is preventing you from getting it using normal avenues).

A good solution for getting the data from the system would be to insert a USB stick and then click on a script button that would copy all the extractions to the stick and afterwards delete them from the SD card (but leave the data in the database untouched).

I think that is the best solution. I had been working on a script that auto-mounted USB sticks and other external storage devices to the actual $RECS_DIR directory, but this makes more sense and is much easier to implement. There is still a huge obstacle in this development, though, which is that a filesystem type that would work for Windows, Linux and Mac (FAT32) doesn't support symbolic links. Since all content in the By_Common_Name and By_Scientific_Name directories are symbolic links, this means that a script buttons' best effort in copying the data to an external storage device would to copy the By_Date directory only. That will need to suffice for the time being.

Maybe you must could implement a stop script that prevents overwriting of the system when the SD card is full of recordings? That would help prevent unpleasant surprises ;-).

This is a must. I've not yet had the luxury of filling a system up with data, and am really surprised that @CaiusX's installation still has 16GiB of storage left after so long and so many detections! I suspect the easiest thing to do would be to have a small script that checks disk space and when it reaches, say, 95%, it will call stop_core_services.sh so that things stay online for file transfer but the data collection is stopped.

Just curious, did this installation have the customized cleanup cron job (to keep 2 days of data) or the default cron?

[DD4WH]

Patrick, thanks a lot for your detailed answer, that makes it a lot easier to understand the things I observe. I used another RPi to read the SD card contents of the full SD card with an SD card reader and transfer those to a USB stick with a Windows-formatted file system. And I observed exactly what you described, there were no files visible on a Windows PC, although the RPi copied about 50Gigabytes to the stick. Also on the stick -when inserted on the RPi- the file manager said something like "cannot read symbolic links". That is why I thought the OS/file system would be totally broken in general.

But now I understood that the files are "physically" present only in the "extracted" directory (which cannot be read by Windows, even if copied to a Windows USB stick) and the other "files" in the other directories are not present in the directories, but are "symbolic links".

But that also means that there is no way for me to get hold of the recordings? How can I get the files onto my USB stick, which is Windows-formatted, so I can read and copy the recordings to my Windows PC system for further analysis? I cannot use network transfer, because the system runs either totally unconnected to any network or runs through an LTE router with very limited network speed. Would some external software work that runs on my Windows PC and can actually read a USB stick that has files on it that have been copied from the RPi to a Windows-formatted stick? Or would I have to format the stick in Linux format (how could I do that?) and then plug into the Windows PC and start the software to transfer? Oh, I see I lack a lot of knowledge in that respect. I will do a thorough research on the web to make these things clearer for me and to find a way to get hold of the recordings that BirdNETPi produces.

And, Yes, you are totally right! I used a customized cron job that keeps ALL the recordings of the last two days and that would probably be one of the reasons why my card filled up so quickly.

[DD4WH]

OK! I was now successful in reading the files with Linux Reader. --> 57Gigabyte of Audio :-) Thats nice to be able to recover the data! However, one thing I realized was that many extracted recordings were deleted! In every species folder there are only 20 recordings left and the older recordings are deleted (which is a pity, because all the older Pygmy owl recordings became deleted because of one recent morning with high activity . . .). I can imagine this is intended to save SD space. But I would need to change that for my purposes. Maybe I will make a structured list of "specific parametres" with the locations, which scripts have to be modified, so I can make the system optimal for specific purposes.

UPDATE: I was wrong --> only the entries (symbolic links) seem to be restricted to 20 entries, the recordings seem to be there in the folder ("by date"). Thanks Patrick, for your extensive commenting in the code! This makes it possible to understand whats going on --> found that hint in extract_new_birdsounds.sh

[mcguirepr89]

UPDATE: I was wrong --> only the entries (symbolic links) seem to be restricted to 20 entries, the recordings seem to be there in the folder ("by date"). Thanks Patrick, for your extensive commenting in the code! This makes it possible to understand whats going on --> found that hint in extract_new_birdsounds.sh

So glad you noticed, I was just drafting a little note about that to you.

Also, I just saw your post on the Sharing Stuff discussion and I am so proud to have been a part of that awesome research. Thank you!

[mcguirepr89]

Maybe you must could implement a stop script that prevents overwriting of the system when the SD card is full of recordings? That would help prevent unpleasant surprises ;-).

This is fixed now https://github.com/mcguirepr89/BirdNET-Pi/commit/dc0f8aff5d2c0d5801625eca60afacb63ee72ad2

Thank you for being a tried and true tester :1st_place_medal:

[mcguirepr89]

Working on this today

[mcguirepr89]

Update: Thanks to the SQLite version, the issues surrounding the By_Common and By_Scientific symbolic linking are no longer a problem. As a result, this goal may now be within reach. I still don't have a USB thumb drive to test with, but I don't think that will completely prevent me from developing something around this.

[CaiusX]

Reclosed :)