Feature Request - Retention by Available Disk Space

[XtremeOwnageDotCom]

Reading through the documentation, and the available configuration parameters-

I do not see the ability to say, allocate 3TB to recordings.

In a typical NVR setup, you would typically dedicate a single, large disk to your NVR system. In my current setup, I use a 8TB disk in conjunction with Blue Iris.

I can set specific retention times on a per camera/per folder basis if required, but, for my recordings, I would prefer to maximize my retention based on the amount of available disk space I have. When the disk starts filling up, delete the oldest recordings to free up space.

TLDR;

Asking for a new feature to control retention by disk space, rather then only by a static number of days.

[rhatguy]

+1 for this. Many use cases would want to just keep as much as possible in a given amount of space.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[XtremeOwnageDotCom]

• Further Activity. Still a REALLY good.... idea to add.

[OptimusGREEN]

yeah this would be super handy. My feeds have actually stopped showing in home assistant and after some debugging it appears to me my nvr drive is full.

[2021-11-07 18:59:33] ffmpeg.driveway.record_rtmp ERROR : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device

[Bod1985]

This would be really useful. I use docker inside a Debian VM with a second hard drive passed through specifically for recordings.

This drive became full today causing IO errors and crashed my entire Debian VM. (I can make improvements elsewhere to prevent this in future but it also highlighted my desire for this feature in Frigate)

[mnaz]

I would agree it would be nice to have a setting to help protect the amount stored to avoid filling the drive accidentally.

[bagobones]

Doing this with the option for fixed space used, % of available disk, or minimum free space to keep would be ideal.. Not all NVRs implement all 3 options but most have some form of roll over based on space. These options are particularly useful if you are storing on a shared NAS where fixed max might be best, or if you just want to target a disk and leave a little space free.

[Vitani]

I'm using this Powershell script, run on a 15 minute schedule, to tidy up my recordings, hope it helps someone else (just change the $drive, $target and $desiredGB variables). Obviously I do not provide any warranty, and use at your own risk.

$drive = "D:"
$target = "$drive\Frigate\Media\recordings"
$desiredGB = 100 * 1GB

$freeSpace = ([wmi]"Win32_LogicalDisk.DeviceID='$drive'").FreeSpace
if ($freeSpace -ge $desiredGB) {
    Write-Host "Free space is $([math]::Round($freeSpace / 1GB))GB/$([math]::Round($desiredGB / 1GB))GB. Nothing to do."
    exit
}

$spaceNeeded = $desiredGB - $freeSpace
Write-Host "Need to free up $([math]::Round($spaceNeeded / 1GB))GB, Enumerating files ..."

$files = Get-ChildItem $target -File -Recurse | Sort-Object CreationTime
$spaceSaved = 0mb
$filesToDelete = foreach ($file in $files) {
    $file #return the file so it gets added to $filesToDelete
    $spaceSaved += $file.length
    if ($spaceSaved -ge $spaceNeeded) {
        break
    }
}
if ($filesToDelete.Count -eq 0) {
    Write-Warning "Done; No files available to delete. Further action required."    
}
else {
    Write-Host "Done. Deleting $($filesToDelete.Count) files ..."
    $filesToDelete | Remove-Item -Force -Confirm:$false -Verbose

    $tailRecursion = {
        param(
            $Path
        )
        foreach ($childDirectory in Get-ChildItem -Force -LiteralPath $Path -Directory) {
            & $tailRecursion -Path $childDirectory.FullName
        }
        $currentChildren = Get-ChildItem -Force -LiteralPath $Path
        $isEmpty = $currentChildren -eq $null
        if ($isEmpty) {
            Write-Verbose "Removing empty folder at path '${Path}'." -Verbose
            Remove-Item -Force -LiteralPath $Path
        }
    }   

    Write-Host "Done, deleting empty folders ..."
    & $tailRecursion -Path $target

    Write-Host "Done, waiting 30 seconds before checking free space again ..."
    Start-Sleep -Seconds 30
    $freeSpace = ([wmi]"Win32_LogicalDisk.DeviceID='$drive'").FreeSpace
    if ($freeSpace -lt $desiredGB) {
        Write-Warning ("Files deleted only freed {0}GB of {1}GB needed. Further action required." -f [math]::Round($spaceSaved / 1GB), [math]::Round($spaceNeeded / 1GB))
    }
    else {
        Write-Host "All done, $([math]::Round($freeSpace / 1GB))GB of free space available."
    }
}

[einarhauks]

Please either implement this or allow us to configure some maximum space for Frigate recordings. When running Frigate as an Add-On in Home Assistant OS, one cannot simply assign recordings to another volume (please correct me if I'm wrong). Having Frigate filling up the Hassio disk is a real pain...

[NPutting]

I'm using this Powershell script, run on a 15 minute schedule, to tidy up my recordings, hope it helps someone else (just change the $drive, $target and $desiredGB variables). Obviously I do not provide any warranty, and use at your own risk.

$drive = "D:"
$target = "$drive\Frigate\Media\recordings"
$desiredGB = 100 * 1GB

$freeSpace = ([wmi]"Win32_LogicalDisk.DeviceID='$drive'").FreeSpace
if ($freeSpace -ge $desiredGB) {
  Write-Host "Free space is $([math]::Round($freeSpace / 1GB))GB/$([math]::Round($desiredGB / 1GB))GB. Nothing to do."
  exit
}

$spaceNeeded = $desiredGB - $freeSpace
Write-Host "Need to free up $([math]::Round($spaceNeeded / 1GB))GB, Enumerating files ..."

$files = Get-ChildItem $target -File -Recurse | Sort-Object CreationTime
$spaceSaved = 0mb
$filesToDelete = foreach ($file in $files) {
  $file #return the file so it gets added to $filesToDelete
  $spaceSaved += $file.length
  if ($spaceSaved -ge $spaceNeeded) {
      break
  }
}
if ($filesToDelete.Count -eq 0) {
  Write-Warning "Done; No files available to delete. Further action required."    
}
else {
  Write-Host "Done. Deleting $($filesToDelete.Count) files ..."
  $filesToDelete | Remove-Item -Force -Confirm:$false -Verbose

  $tailRecursion = {
      param(
          $Path
      )
      foreach ($childDirectory in Get-ChildItem -Force -LiteralPath $Path -Directory) {
          & $tailRecursion -Path $childDirectory.FullName
      }
      $currentChildren = Get-ChildItem -Force -LiteralPath $Path
      $isEmpty = $currentChildren -eq $null
      if ($isEmpty) {
          Write-Verbose "Removing empty folder at path '${Path}'." -Verbose
          Remove-Item -Force -LiteralPath $Path
      }
  }   

  Write-Host "Done, deleting empty folders ..."
  & $tailRecursion -Path $target

  Write-Host "Done, waiting 30 seconds before checking free space again ..."
  Start-Sleep -Seconds 30
  $freeSpace = ([wmi]"Win32_LogicalDisk.DeviceID='$drive'").FreeSpace
  if ($freeSpace -lt $desiredGB) {
      Write-Warning ("Files deleted only freed {0}GB of {1}GB needed. Further action required." -f [math]::Round($spaceSaved / 1GB), [math]::Round($spaceNeeded / 1GB))
  }
  else {
      Write-Host "All done, $([math]::Round($freeSpace / 1GB))GB of free space available."
  }
}

Do you have any problems with the DB becoming bloated when manually deleting the recordings?

[Vitani]

Do you have any problems with the DB becoming bloated when manually deleting the recordings?

I didn't notice, but I have since moved to Blue Iris as I found it more reliable & had more features that I wanted to use.

[tehniemer]

In my mind this is a very necessary feature for those who keep 24/7 recordings. It would be much easier to set a storage limit than try to guess how many days it'll take to fill it up. In this case it would also make adding or removing cameras painless.

[ozett]

as advised i wil add my wish to this FR here, that an infomation about remaining recording-time would be great.

that would be highly dynamic, based on recent savings, deletion or blocked-deletions (favorites clips become delete-protected). but some AI could kick in to make this estimations about remainig time, based on actual disc space - and historical data. see https://github.com/blakeblackshear/frigate/pull/3225#issuecomment-1133064092

[elektrinis]

Frigate crashed my whole smart home while I was away on vacation, due to disk being full. I can't believe there is no disk-related configuration, like to use full disk and set the location, at minimum. "Retention" setting is really not as useful, as free space is the limiting factor in most cases.

[NickM-27]

"Retention" setting is really not as useful, as free space is the limiting factor in most cases.

Segment sizes are quite consistent / predictable so figuring out how many days = GB is fairly easy to do.

That being said, the flexibility and convenience is recognized which is why this FR is pinned and will be included in a future release (hopefully 0.12 but we'll see)

[XtremeOwnageDotCom]

"Retention" setting is really not as useful, as free space is the limiting factor in most cases.

Segment sizes are quite consistent / predictable so figuring out how many days = GB is fairly easy to do.

The problem comes in, when you are storing snapshots, segments, and other activity which wouldn't be factored into that math. As well, if you are like myself, and you have dozens of cameras to manage, it becomes much more difficult to configure this properly. Lastly, most of my recordings are saved straight to disk, without any transcoding to save on the limited CPU available for my server. Given I have multiple different manufacturers of Cameras (Reolink, Hikvision, Amcrest, etc....), the encoding/resolution/bitrate will not be consistent across cameras. Some are 5MP / H265, others are lower resolution using H264, for example.

So, while determining 'days = GB' may be quite feasible if you are only running one or two cameras, this becomes significantly more difficult when you add many cameras, of varying manufacturer/capability.... and even more difficult when you factor in space being used for "other data" such as other clips, snapshots, etc.

[bagobones]

Segment sizes are quite consistent / predictable so figuring out how many days = GB is fairly easy to do.

If this statement was true then the original feature request would be trivial.

As others pointed out it is not due to all the other disk using features.

Simply we need the recorder to realize space is about to run out and start purging / rolling over old content when a set amount of free space is not available.

[elektrinis]

This ^

[NickM-27]

Given I have multiple different manufacturers of Cameras (Reolink, Hikvision, Amcrest, etc....), the encoding/resolution/bitrate will not be consistent across cameras. Some are 5MP / H265, others are lower resolution using H264, for example.

To be clear, I wasn't saying it was a great solution but suggesting how the "days" option can be utilized for storage retention until this is supported. The segment size for each camera will be consistent, so even if one is h265 and one is h264 and they are different, each individual camera will have predictable sizes segment to segment.

If this statement was true then the original feature request would be trivial.

As others pointed out it is not due to all the other disk using features.

It is true, see this comment for how to do it with your own camera.

Also see this comment for a partial explanation for why this hasn't been done "ASAP".

(to quote it)

This feature will probably be slated for the 0.12.0 release. I am happy to help you calculate the 
appropriate number of days to configure to ensure you do not exceed available disk space again.
I would need to to know the approximate size of each of the 10s video segments for your cameras
in the recordings folder.

I realize this feature is important to you and likely several other users. Every user believes their request 
is the most important, and I am doing my best to triage requests and work on them as they come up. 
I have been running Frigate at several locations for many years and never once lost any footage 
with an appropriate setting for retention days. It's not automatic or ideal, but video segment sizes 
are very predictable and you can determine how much space each day will consume. This has been sufficient 
for the overwhelming majority of users. If Frigate's current feature set does not meet your needs, 
perhaps it is not the right solution for you.

It is not trivial to implement due to a large number of edge cases and considerations for setups with many cameras, users who only want some cameras to be storage based, etc.

I love the application but stop building silly features and saying "hopefully on 0.12".

You noted in your previous issues that you do not use Frigate's AI features / object detection at all. So calling the features that have been worked on "silly" because they do not apply to your particular use case is disingenuous.

Frigate has many users with many different use cases / priorities, we have to do our best to cater to everyone as each person believes their feature is "top priority". I said "hopefully" as I am not in a position to promise anything for the next release cycle, but it is definitely a high priority feature that is recognized.

[esmoyer]

People do realize that this is not the only NVR software available, right? Frigate is free and developed by Blake and others giving their time to provide us with an NVR option with some pretty awesome features. I can't believe how far it has come.

I get putting in a feature request or idea, but, unless I'm interpreting things wrong, reading through some of these comments (leaning more towards demands), it sounds like some of you need to be shelling out $$$ for a commercial or enterprise system instead. Frigate just might not be for those of you who need or expect it to have every little x, y, z feature.

As some of these "requests" start turning into "demands", I sure hope y'all are making good use of that Donate/Sponsor link. Or, better yet, if it's so simple, fork the code, implement the feature yourself and submit a pull request to be merged.

[blakeblackshear]

I don't think there is much discussion left on this request that helps clarify how this should be implemented. I hear you same as I hear everyone else. No need to keep advocating for this other than a :+1: on the original post.

[NickM-27]

closing as the original request for frigate cleaning up recordings when storage is full has been implemented since 0.12, subsequent more specific feature requests can be made.