Purge older data more effectively

[markmac99]

RMS' DeleteOldObservations deletes older data to free up space for the next night's run. The logic is to first delete the oldest CapturedFiles folder (CF) , then if thats insufficient to delete the oldest ArchivedFiles folder (AF), and repeat this pattern till sufficient space is freed. Once space is freed, RMS will create one new CF and one new AF for the current night's data. Importantly, if all CF have been deleted, and there's still not enough space, then RMS aborts.

However, AF folders are much smaller than CF folders (typically < 1GB vs 20GB). So there are two scenarios: a) deleting one CF frees enough space. The number of CF remains constant and one new AF is created every night. b) deleting one CF is not sufficient, so one AF is deleted as well, followed by another CF. The number of CF drops by one, but the number of AF remains fixed.

Note that the number of AF never goes down. This is because the logic is to delete one CF, then one AF if and only if needed, so on most nights, no AF are removed.

Consequently, over time, the number of AF folders increases while the number of CF remains constant or falls. Eventually RMS has no more CF folders to delete, and will now stop data capture, even though it could actually free space by deleting old AF folders. Furthermore the AF folders are made into a compressed Tar archive, consuming a bit more space that is never reclaimed.

We propose therefore to put in a limit on the number of AF and BZ files, and to automatically delete the oldest. The limit will be user-definable in the config file

[markmac99]

My proposal is to add two optional parameters to the config file in the "Capture" section arch_dirs_to_keep: default 30 bz_files_to_keep: default 5 in both cases, zero means keep everything. This is the current RMS behaviour, but flawed as noted above.

[satmonkey]

I know this is not directly related to current issue, but could you change the extra_space_gb parametr to 25 as default? A single night during the winter consumes ~ 20gb + some extra space for GenerateTimelapse is needed.

[markmac99]

RMS already takes into account the space required for the next night's run, so last night it worked out i needed 19GB plus the extra-space which i have set to 6GB. So i think there's enough already being set aside. The timelapse is only 20MB or so, and should not incur too much extra. I do think a default of 6GB would make sense tho.

Also see my comment on the bug in the use of ephem. I'll fix this, which i suspect is causing the routine to fail silently in many cases.

[markmac99]

ephem bug is here:

        # Initialize the observer and find the time of next noon
        o = ephem.Observer()  
        o.lat = config.latitude 
        o.long = config.longitude

values passed to ephem() should either be in radians, or should be string values in degrees eg "12.345". ephem is misinterpreting a value of 12.345 as being in radians, and normalising it to the range [-pi/2,pi/2] so calculating 1.349 radians, a latitude of 77 degrees north!

[satmonkey]

I got the point now, so this is the extra space in addition to the space calculated. However, I did a fresh install last week using standard 64bit bullseye image and having 5 gb in .config as default on the 128 gb card, it ran out of space after several days. It slowed down, the timrlapse creation did not finish in time before the next run started. As for the timelapse needs, take into account the temp files created, each of them is I believe ~400k. And I suspect ffmpeg to need additional buffer space for the video itself.

[markmac99]

i think you've hit the other bug which was preventing the deleteOldObservations routine from running at all - should be fixed by my recent pull req which denis just merged. UKMON have been creating a timelapse every night for a while without problems, but we do make sure the Pi is rebooted every morning. I think this helps many processes as it flushes out memory buffers and frees up any leaks. We also set extra-gb to 6 just to be sure. However, I do also monitor space usage every 5 minutes to my OpenHAB dashboard so i can go back and check how much was consumed by the timelapse.