Maximum disk usage

[GrossoMoreira]

Hi,

Today I noticed my disk was full with kerberos io video files.

Is there a way to limit the disk usage? This would be really helpful.

Thank you!

[shacal]

Quick and dirty: CRON daily: find /DATAROOT/capture -type f -mtime +14 -name '.' -execdir rm -- '{}' \; Remove older than 14 days files.

[GrossoMoreira]

I thought of that too, but that's not a good solution because:

1. It assumes that disk usage is linear with time, when in reality there are days when there is much more activity than others;
2. This script does not guarante that the disk usage doesn't go above the desired limit. Even if the script used the du command or something like that to check the total disk usage of all video files and delete files while it's above the limit, it would only do so after each video file was created, in which case it would be possible that the disk usage would go above the desired limit even if only for a short period of time (the time after the last file was written, before the script would run and delete other files).

So, the good solution is for kerberos io to check the disk usage while the video file is being written to disk, and delete older files in case the current file would surpass the limit. This can only be done by kerberos itself (and not by an external script) because only kerberos knows how much data is going to be written in each file before actually writting it to disk.

[shacal]

Sure, but it is security, so you should be more concern on dates than space. Let's say 14 days, two weeks... how long it will take, to notice something is missing/broken. If 14 days is enough you can find your suspects... if not, use 30 days. And, if you are concerned about your space, and you limit it let's say to 1tb, that might mean, that heavy activity in front of camera, erases images of event that happened an week ago...

So, based on this key concept, days before event, there is zero reason to have limit factor in surveillance system.

[GrossoMoreira]

I think it makes sense to me, in my particular case. I have reasonably limited storage (150GB) and I am not worried about what happened a week ago. I mostly just want to see the images immediately after movement is detected. That said, it would still be nice to keep as much footage as possible as long as it doesn't go over the 150GB limit.

However, although I believe my necessity is legitimate, maybe my case is just too particular/specific to expect kerberos io to support it.

[cedricve]

hey @GrossoMoreira, just wondering how did you installed Kerberos.io?

[GrossoMoreira]

In docker. I also searched for ways to limit the disk usage of existing containers, or using docker-compose, but it seems like it's quite complicated.

And even if I were able to limit disk usage via docker, whould kerberos io automatically delete old files to make space for new ones?

[shacal]

If you limit with filesystem, kerberos would just fail to write.

[GrossoMoreira]

Well, that's disappointing.

[cedricve]

Depending on how you installed Kerberos.io, it will start deleting old media when disking is getting full. On KiOS we have a init script running: https://github.com/kerberos-io/kios/blob/master/board/common/overlay/etc/init.d/S85machinery#L38-L48, on docker we have a run script: https://github.com/kerberos-io/machinery/blob/develop/docker/run.sh#L7.

However after reading your requirement that makes sense, but might be difficult because we should give Kerberos.io context aware of all the different environments it can run on. What we do at this moment is to make sure the disk usage isn't above 90% procent, so the chance that another 10% of disk will be consumed while Kerberos.io is recording is rare. Hope this makes sense guys..

[meeki007]

I've got 60+ containers running on a dell server. I have a drive pool of 87TB. I've noticed that in time one camera container will beat the others to deleting the files after 90% is hit. So i end up with 2 months of video for one container and 4+ for another. Adding another camera can be fun too.

Ive resorted to using a script that goes through all the mounted container folders and picks the oldest files and removes them at 88% drive space. Beating the runny.sh script in the containers to the punch.

NOTE: the runny.sh script is no longer run.sh see https://github.com/kerberos-io/docker/blob/master/Dockerfile

Merge the two run files.

ADD ./run.sh /runny.sh RUN chmod 755 /runny.sh RUN chmod +x /runny.sh RUN sed -i -e 's/\r$//' /runny.sh

[cedricve]

@meeki007 are you interested in sharing what you have developed?

[meeki007]

Oh nothing much. Its all just hacked together as people found ways to break the system.

Got 11 people using the system and some funny guy/girl Clicked the Remove Images Button in the System section. Never found who did it and lost 4 months of video. Solution: SystemController.php comment out the function 'cleanImages()' now they can click it all day it aint doing it.

Just look at how i start a container for a camera and you can see Ive cobbled it together. docker run -it --restart always --log-opt max-size=5m --log-opt max-file=10 \ -v /home/sun/cameras/SunValleyOutdoorTruckYard03/config:/etc/opt/kerberosio/config \ -v /home/sun/cameras/SunValleyOutdoorTruckYard03/webconfig:/var/www/web/config \ -v /home/sun/cameras/SunValleyOutdoorTruckYard03/logs:/etc/opt/kerberosio/logs \ -v /home/sun/cameras/SunValleyOutdoorTruckYard03/controllers:/var/www/web/app/Http/Controllers \ -v /home/sun/cameras/SunValleyOutdoorTruckYard03/drivespace_script/runny.sh:/runny.sh \ -v /mnt/virt/cameras/SunValleyOutdoorTruckYard03/capture:/etc/opt/kerberosio/capture \ --name SunValleyOutdoorTruckYard03 -p :80 -p :8889 -d kerberos/kerberos

as you can see I'm just editing the hell out of things to make it work with out using the cloud :) Then I tie it all together with a map.

You click a camera and then you get a preview image and then you can go live view or review the video.

I also use node red to do allot of getting camera streams from old jpg cameras into a mpeg stream then kerberos.io can record it.

So would I like to contribute????? Sure but most the stuff I did would be to avoid the cloud. I have not published my stuff because I think Its wrong to step on your toes with making a buck while providing a way more polished service than I have out of the box.

My company wanted 6 months of recordings in High Resolution and I dont think we can afford your cloud service for that :)

[meeki007]

I'll Write up a github Markdown How-to next weekend on cleaning up Docker Containers and the runny.sh script I use. I'll also add a how-to on using node-red to publish a mpeg stream from hard to connect cameras so others can connect them up to kerberos.io