search

blakeblackshear / frigate

NVR with realtime local object detection for IP cameras
https://frigate.video
MIT License
18.24k stars 1.66k forks source link

archive storage #3673

Open toxic0berliner opened 2 years ago

toxic0berliner commented 2 years ago

I want to be able to move old recordings to my NAS so that I can have recent recordings on my local fast SSD but still keep older footage available on my NAS having terabytes of spinning rust available.

Describe the solution you'd like A way to configure an archive storage (either a different path that I mount using NFS or smb or even directly the connection parameters for frigate to use SMB client himself)

Describe alternatives you've considered No real idea, except running frigate twice and keeping duplicate of the "main" storage.

Additional context Having 2 level of storage is the key here, moving data from main to archive storage can be complicated within frigate, so a first version could be to add a "cache" storage that fully duplicate the main storage but allows to setup shorter retention. This way I could set retention to 90 days on main storage and point it to a NFS mount and add a "cache" retention of 7 days in the local SSD under /tmp. This way it could be less complicated to implement: any recording in the db is available in main storage (NFS) if it was not found in the "cache" (SSD or /tmp)

rhatguy commented 2 years ago

Just thinking...could something like this be created with UnionFS and a separate script that moved all files over X days old to the "archive" filesystem. I believe UnionFS would allow frigate to still see both drives as a continuous filesystem regardless of which disk the files actually live on.

https://en.wikipedia.org/wiki/UnionFS

NickM-27 commented 2 years ago

yeah, I don't think this is something that makes sense for frigate to integrate itself. For example, my Unraid NAS already does this as the default behavior. There are shares which consist of one or many harddrives, and all writes to a share are done to a cache drive first and those files are moved to the harddrive when the cache gets full. Frigate just sees this as one big drive.

We don't want frigate managing the native filesystems, we want one volume mount for recordings without caring what the actual file system is behind that.

toxic0berliner commented 2 years ago

Seems a bit of a hack but could be possible. Unionfs seems quite old though or at least esoteric... Given frigate has to manage massive volume of data I would expect archiving or caching strategies to fit many people's requirements but I might be mistaken.

toxic0berliner commented 2 years ago

I get that it's tempting to rely on the intelligent FS we have nowadays. But let's be honest too it will probably not be fine grained with a retention duration set in frigate, even people using zfs and cache layers might want to remove the recordings before the cache is actually full to keep some for other uses...

Managing storage seems like a must for me for a recorder of my IP cameras. If this is not in the future of frigate I'll have to look someplace else.

It's not directly linked to this feature but consider that most NVR implement a rotation of recordings when the drive is full as one of the first feature, so managing the storage does make sense for an NVR

NickM-27 commented 2 years ago

This can also be done with zfs which is a modern file system: https://unix.stackexchange.com/questions/316422/ssd-as-cache-for-filesystem-e-g-on-hdd-or-any-other

Other options as well: https://blog.delouw.ch/2020/01/29/using-lvm-cache-for-storage-tiering/

NickM-27 commented 2 years ago

I agree it would be useful, as I use it myself, but I don't think it's something that makes sense for frigate to re-implement as many solutions already exist and we have many users already doing this.

NickM-27 commented 2 years ago

It's not directly linked to this feature but consider that most NVR implement a rotation of recordings when the drive is full as one of the first feature, so managing the storage does make sense for an NVR

This is already an existing feature request that is going to be implemented in #994

Touching multiple drives / volumes and managing all of that is a whole other level that I don't think it makes sense for frigate to manage itself.

That being said, I just speak for myself and it's ultimately not my decision so it could still be decided to be done at some point.

rhatguy commented 2 years ago

@toxic0berliner I'm not disagreeing with your feature request, but I think where this discussion really begins to be helpful is when you talk about tiering off to a cloud provider (probably via object). I suspect many people would be interested in keeping 7 days onsite, then tiering off to S3 or backblaze. Technically this could be achieved with a similar UnionFS type approach. I feel like this really boils down to whether the frigate developers think the app needs to be intelligent enough to tier data, or whether they are going to leave that up to lower layers. My personal opinion would be that there are more beneficial features to implement than tiering storage when there are other ways to accomplish tiering.

toxic0berliner commented 2 years ago

Not disagreeing with you either 😀 Storage tiering makes sense I believe but I could fully understand other priorities might take precedence. As well as what kind of tier to add if object or block can be debated. I have little doubt that it's too complicated for my level of dev anyway so I'll be patient 😜

javydekoning commented 1 year ago

I disagree that this should be left to the Filesystem because that limits you to a single FS (type). This also brings un-needed complexity.

It's very common for container orchestrators to have different storage classes that are easy for the user to leverage.

For example:

"Hot" mounted at: /media/frigate/recordings/hot "Cold" mounted at: /media/frigate/recordings/cold

This allows recent recordings to be local (block/nvme/ramdisk/whatever) while the rest is moved to remote NFS for example. The only thing frigate would have to do is mv the object using a tiering policy (e.g. 7d -> mv cold, 1y -> delete).

Implementing s3 compatible storage at a later point in time would be a nice to have as well. That would allow users to keep recordings at the cloud provider of their choice (durable and cost effective).

sstratoti commented 1 year ago

Just sharing my experience/use case. I've taken the plunge with Frigate off of Blue Iris, and so far the experience has been excellent. Especially with the .12 release! But in my previous setup, Blue Iris allowed me to set a size limit or an age limit for a directory of storage.

It also allowed me the options to either delete or move older files to another directory as it reached that limit.

It would run a maintenance job every few minutes (let's say 10, not actually sure) to move/rotate the files out or delete them.

The same settings could be set for the "other directory". Either age or max size.

This allows you to manage the maximum space that the app should use so you can leave other space on your main drive for other applications, docker containers, files, swap, etc.

It would also allow me to keep the most recent snapshots and videos on my SSD, and move older files that I'd like to keep, but probably not access, in a slower USB 3.0 SATA external drive that is much larger.

I could move everything over to the SATA drive, but then I'll have like 990 GB of SSD drive that'll just be sitting there unused. I could copy/archive things myself using rsync or something, but then I'd lose access to them in the timeline once the main files get removed by the new .12 file rotating feature.

Unlesssss...if I move the files myself, would it be possible to update the SQLite DB to point at the new location of the files so that they aren't lost in the UI timeline? Are those entries cleaned up somehow?

rhatguy commented 1 year ago

Not arguing whether this approach is the "best" way to achieve what you want, BUT...if you're willing to move the files yourself you can easily use unionfs today to combine /ssd_mountpoint and /hdd_mountpoint into a new mountpoint /frigate_storage. Then you can move files in the background manually and still have everything show up to frigate. Frigate doesn't know which underlying filesystem the files live on. No need to update the database. Something like the below should accomplish what you want. Then you just need to move the files manually from /ssd_mountpoint to /hdd_mountpoint at whatever criteria you decide.

mount -t unionfs -o dirs=/ssd_montpoint:/hdd_mountpoint=ro none /frigate_storage

NickM-27 commented 1 year ago

I run in unraid which does this automatically, it uses fuse.fs

kravemir commented 11 months ago

I want to be able to move old recordings to my NAS so that I can have recent recordings on my local fast SSD but still keep older footage available on my NAS having terabytes of spinning rust available.

This is already a solution, which is to "move" recordings, and not a problem statement, of what do you want to solve in real life.

Real life problems statement might be:

However, when this is combined with other environmental constraints:

Depending on environmental constraints,

In combination with above, when on doesn't do constant remote streaming/storing:

I wouldn't focus on how to solve things, as long as it solves real life problems in given environmental constrains, then I would just go with the easiest and/or cheapest available solution.

The constraint is something, that is not in power of owner to change, i.e. not that I want to do it this way. But, that in my location, I don't have any unmetered connection available.

sstratoti commented 11 months ago

My issue isn't that the files need to be moved. It's that once they're moved you can't access them within the UI to look for/at past events.

When using unionfs or mergefs, my understanding is that it fills up one disk and then moves onto the next while combining it to look like a single directory.

Doing it this way makes sense if you have multiple HDs that you want to combine into one, but if I want to utilize my SSD/NVMe drive for writes and then spinners for storage/archive, is there a way to do this unless Frigate controls which disk/folder it writes to and reads from?

If it sees it as a single directory, and FIFOs the files, would it maximize the benefit of the SSD?

I'm totally up for trying this method if my understanding of this is incorrect. Thanks!

NickM-27 commented 11 months ago

but if I want to utilize my SSD/NVMe drive for writes and then spinners for storage/archive, is there a way to do this unless Frigate controls which disk/folder it writes to and reads from?

Yes, this is exactly what unraid does via fuse.fs and it works perfectly. It appears as one volume to frigate, and the OS decides when to move files from SSD to the HDD archival storage

sstratoti commented 11 months ago

And this works if I'm not using unraid?

NickM-27 commented 11 months ago

It's nothing proprietary to unraid, though unraid makes it really easy

sstratoti commented 11 months ago

Alright, I'll give it a try. Thank you!

toxic0berliner commented 11 months ago

Regardless, this is a feature request and the proposed "real life problem statement" doesn't match neither my needs nor that of others. Storage tiering is implemented in several NVR solutions and for good reasons, it can then be used in many ways to achieve whatever the user can think of building with it.

On my side I got it working with unions with a /hot that is being written to by default and a custom script+cron that moves files to /cold.

Nonetheless I feel this is not a proper solution and I still hope tiering will someday come to frigate. It has the DB and moving files shouldn't be too tricky, options for retention duration on each tear would benefit from being integrated as well as storage statistics. On my little ssd it took me a while to find how old files should be before I move them, doing the logic for moving files until drive is 80% full starting from the oldest files was too tricky for me to di in bash and not a long term solution I believe.

javydekoning commented 11 months ago

I want to be able to move old recordings to my NAS so that I can have recent recordings on my local fast SSD but still keep older footage available on my NAS having terabytes of spinning rust available.

This is already a solution, which is to "move" recordings, and not a problem statement, of what do you want to solve in real life.

Real life problems statement might be:

  • have recent recordings available as proof of a burglary or an incident:

    • what's point of recording if not to have a proof?

    • in case of physical breach, the proof of burglary should not be stored only on just one local device, which increases probability of proof material being destroyed or stolen during burglary,

  • have recent recordings available on SSD for fast browsing,

  • have old recordings available on HDD on NAS for archivation:

    • cover very low probability of a need to dig into those in future.

However, when this is combined with other environmental constraints:

  • solution is deployed in house/SOHO/facility with metered LTE or 5G connection, and it's not possible to constantly stream out camera recordings.

Depending on environmental constraints,

  • one might stream out camera footage to remote frigate instance (via VPN),

  • one might record simultaneously to multiple storage devices via single frigate instance:

    • keep duration is low for SSD device,

    • keep duration is high for HDD device,

  • one might record simultaneously via multiple frigate instances:

    • primary instance has got all cool features (detection, browsing, events,..) enabled,

    • secondary (archival) instance has is only storing camera streams to HDD, which has low requirements on CPU/RAM.

In combination with above, when on doesn't do constant remote streaming/storing:

  • primary instance could be always sending only events out (JPEG, parts of stream,...) to remote storage,

  • primary instance could be sending only events in case it's turned on (i.e. away from house mode) to save data on metered connection - when people are present at site/home:

    • storage devices aren't unattended,

    • people generate much more events, and that increases amount of data sent to remote storage,

    • also these extra events aren't important to be saved on remote.

I wouldn't focus on how to solve things, as long as it solves real life problems in given environmental constrains, then I would just go with the easiest and/or cheapest available solution.

The constraint is something, that is not in power of owner to change, i.e. not that I want to do it this way. But, that in my location, I don't have any unmetered connection available.

It's manual work. It's very typical to want to store your recordings on remote, durable storage.

Yes I can move it manually but then the Frigate interface won't be able to show them.

The s3 api is widely used and there are many providers offering affordable durable storage. It would be a good option to store recordingsAND metadata.

kravemir commented 10 months ago

It's manual work. It's very typical to want to store your recordings on remote, durable storage.

It's typical to store recordings on remote store simultaneously while storing it locally.

This redundancy is comes useful especially when recording happens to be actually needed, such as burglary event, when the primary NVR might have been stolen in action, and therefore retain the ability to see recordings.

... that I can have recent recordings on my local fast SSD but still keep older footage available on my NAS having terabytes of spinning rust available ...

For me, it makes no sense to implement such "move" functionality.

The most recent recording is often the most precious one. Storing it only on single non-redundant storage is increasing the risk of making security system useless - recordings stolen in action.

So, IMO, storing recordings simultaneously to multiple storages, where each storage has different retention period, is a better alternative, because it provides redundancy.

Yes I can move it manually but then the Frigate interface won't be able to show them.

The s3 api is widely used and there are many providers offering affordable durable storage. It would be a good option to store recordingsAND metadata.

S3 based storage sounds as a viable option. The frigate would see it as one "logical" storage, while the physical storage would be redundant, where SSD and HDD is used for recent recordings, and also tiered, where old recordings deleted from SSD and kept on HDD. This would be transparent to frigate, and flexible - based on S3 storage configuration and capabilities.

The minio can be self-hosted, and there's something about tiering.

Though, not sure whether it would be possible to combine locally hosted S3 with some cloud provider, transparently, where local S3 storage, which is using local low-latency SSD, would work as LRU+write cache for provided remote S3 storage. So, maybe multi-storage support would be needed on frigate level.

fpytloun commented 10 months ago

I just migrated to Frigate from Zoneminder and really miss this feature. My use-case is that I have local storage of limited capacity and then Wasabi S3 mounted as POSIX FS via JuiceFS (k8s PVC). I would like to have option to move records older than x days into this cold storage with longer retention. With ZM it was done simply by having filter actions.

fpytloun commented 10 months ago

Looking at sqlite and it stores path so implementing move script would be simple. Basically just iterate over directories which are by date, move those matching threshold into another location and execute UPDATE statement to update path in sqlite, eg.:

echo "UPDATE recordings SET path = REPLACE(path, '/media/frigate', '/media/archive') WHERE path LIKE '%/2023-11-23/%';" | sqlite3 frigate.db

UPDATE: played a little bit, this should work, just run as a cronjob:

#!/bin/bash -e

FRIGATE_BASEPATH="/media/frigate"
ARCHIVE_BASEPATH="/media/archive"

# Number of previous days to keep
KEEP_DAYS=1

for day in "${FRIGATE_BASEPATH}"/recordings/*; do
    day=$(basename "${day}")
    SKIP=0
    for range in $(seq 0 $KEEP_DAYS); do
        match=$(date -d "-${range} day" '+%Y-%m-%d')
        if [[ $day == "$match" ]]; then
           SKIP=1
           break
        fi
    done

    [[ $SKIP -ne 1 ]] || continue

    echo "[INFO] day=${day} Copying recordings into archive storage"
    cp -ra "${FRIGATE_BASEPATH}/recordings/${day}" "${ARCHIVE_BASEPATH}/recordings/"
    echo "[INFO] day=${day} Updating paths in sqlite"
    echo "UPDATE recordings SET path = REPLACE(path, '${FRIGATE_BASEPATH}', '${ARCHIVE_BASEPATH}') WHERE path LIKE '%/${day}/%';" | sqlite3 "${FRIGATE_BASEPATH}/frigate.db"
    echo "[INFO] day=${day} Deleting original files on hot storage"
    rm -rf "${FRIGATE_BASEPATH}/recordings/${day}"
done

Enjoy 😄

figadore commented 9 months ago

this should work

This looks great, have you tested it? This will still work with all of Frigate's "retain" logic, right? I'd love to see something similar as a built-in feature, but it sounds like this will do the trick until then.

fpytloun commented 9 months ago

this should work

This looks great, have you tested it? This will still work with all of Frigate's "retain" logic, right? I'd love to see something similar as a built-in feature, but it sounds like this will do the trick until then.

Yes, I am using it since sharing and it works nicely. And should work with retain logic as well :-)

WilldabeastHA commented 8 months ago

@fpytloun thank you for sharing this! I have a similar set up where i want to archive after a certain date to spinning disk. Forgive me for my lack of experience with scripts. How do I implement your above script into Frigate. I understand that I will mount /media/archive which will reside on the spinning disk but how do I automate the running of this script?

also will this work in the recent rc release and beta release?

Thanks so much!

fpytloun commented 8 months ago

@WilldabeastHA If you are running in k8s, you can use this CronJob manifest and just tune that volumes part. It assumes that archive.sh is in your configmap which is mounted on /config:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: frigate-archive
  namespace: frigate
spec:
  concurrencyPolicy: Forbid
  schedule: "12 4 * * *"
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      backoffLimit: 0
      # 6 hours deadline
      activeDeadlineSeconds: 21600
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: archive
            image: genunix/sqlite:latest
            imagePullPolicy: Always
            command:
              - /bin/bash
              - -e
              - /config/archive.sh
            volumeMounts:
              - name: frigate-data
                mountPath: /media/frigate
              - name: frigate-archive
                mountPath: /media/archive
              - name: config
                mountPath: /config
          volumes:
            - configMap:
                name: frigate-config
              name: config
            - name: frigate-data
              persistentVolumeClaim:
                claimName: frigate-data
            - name: frigate-archive
              persistentVolumeClaim:
                claimName: frigate-archive
toxic0berliner commented 8 months ago

I also scripted something else based on unionfs and the POSIX creation date of the files to move them from hot to cold storage. I find this less elegant than the given script even if I believe mine can be run hourly instead of daily, but relies on unionfs...

Most important: we are several with the same need and strong enough to justify tweaking stuff on our side...

Would a dev maybe reconsider and take this topic? A few extra settings like archive path and time_before_archival, a scheduled job like the one above, and bingo ?

NickM-27 commented 8 months ago

Would a dev maybe reconsider and take this topic?

what do you mean reconsider? this issue is open and pinned, meaning it will be done at some point. It is however not a priority right now

toxic0berliner commented 8 months ago

My bad, I thought I recalled heavy scepticism in the beginning maybe causing actual devs or maintainers to loose interest. but my memory might be disserving me. Can fully understand priorities srry

PeteB1983 commented 7 months ago

I wanted to add my support for this, and just explain my reasoning as I think it varies slightly from what I have read above.

I have an Unraid server running VM/dockers with a stupid amount of storage (for a normal person - 56TB). however I am a tinkerer and this for various reasons goes offline from time to time.

I would like Frigate to be an always on bullet proof setup - so running on an independent machine with local storage, that then automatically archives to Unraid. Meaning that any maintenance on the main server doesn't bring down Frigate, and it is still capable of making recordings in that time.

the only alteration I would see this as needing to the proposed is either (depending on implementation) a failed transfer system which retries on schedule or when storage becomes available. or if simultaneous storage is used, it would want to save primarily to local storage and them mirror on completion / storage availability to the archive.

I am only adding this so when someone gets to the issue they can consider it, I really appreciate the efforts put in to these things by people much more capable than I.

ckglobalroaming commented 7 months ago

Appreciate this is not a priority but from my perspective most of the above sugestions are boiling the ocean. I have a bear metal frigate with Coral and NVME. I will spin up a second frigate instance in my PVE cluster just for Spinning disk continuous storage. Previous comment regarding other NVR solutions do this out of the box is valid. Look forward to multiple storage tiers becoming reality one day.

Thanks.

WilldabeastHA commented 7 months ago

@ckglobalroaming I just wanna make sure I understand what you’re saying. You have 2 Frigate’s running. you copy/move footage at some point from frigate #1 (NVMe drive) to frigate #2 (spinning disk). Are you just copying the files from one to another and frigate will import/ recognize the file change and in UI it reflects this?

thanks!

alsobrsp commented 6 months ago

@fpytloun This looks perfect for my use case. I am currently writing everything to NFS. Most of the other suggested solutions seem to be targeting local storage. I want the last day or so to be local with the rest remote.

fpytloun commented 5 months ago

Here is new version of my archivation script.

Changes:

    #!/bin/bash -e

    FRIGATE_BASEPATH="/media/frigate"
    ARCHIVE_BASEPATH="/media/archive"

    # Number of previous days to keep (-1 means copy everything)
    KEEP_DAYS=-1

    function retry {
      local retries=$1
      shift

      local count=0
      until "$@"; do
        exit=$?
        wait=$((2 ** $count))
        count=$(($count + 1))
        if [ $count -lt $retries ]; then
          echo "Retry $count/$retries exited $exit, retrying in $wait seconds..."
          sleep $wait
        else
          echo "Retry $count/$retries exited $exit, no more retries left."
          return $exit
        fi
      done
      return 0
    }

    for day in "${FRIGATE_BASEPATH}"/recordings/*; do
        day=$(basename "${day}")
        SKIP=0
        for range in $(seq 0 $KEEP_DAYS); do
            match=$(date -d "-${range} day" '+%Y-%m-%d')
            if [[ $day == "$match" ]]; then
                SKIP=1
                break
            fi
        done

        [[ $SKIP -ne 1 ]] || continue

        echo "[INFO] day=${day} Copying recordings into archive storage"
        find "${FRIGATE_BASEPATH}/recordings/${day}" -type f | while read recording; do
          target_file=${recording/$FRIGATE_BASEPATH/$ARCHIVE_BASEPATH}
          target_dir=$(dirname ${target_file})
          [ -d "${target_dir}" ] || mkdir -p "${target_dir}"

          echo "[INFO] day=${day}, file=${recording} Copying into ${target_file}"
          cp -a "${recording}" "${target_file}"

          echo "[INFO] day=${day}, file=${recording} Updating path in sqlite"
          echo "UPDATE recordings SET path = '${target_file}' WHERE path = '${recording}'" | retry 5 sqlite3 "${FRIGATE_BASEPATH}/frigate.db"

          echo "[INFO] day=${day}, file=${recording} Deleting original file"
          rm -f "${recording}"
        done
    done

    echo "[INFO] Cleaning up empty directories on archive storage"
    find "${ARCHIVE_BASEPATH}" -type d -empty -delete

    echo "[INFO] Cleaning up empty directories on hot storage"
    find "${ARCHIVE_BASEPATH}" -type d -empty -delete
NickM-27 commented 5 months ago

added retry function to bypass database locking. Would not be needed if frigate has option to use WAL journal type

Frigate does use the WAL journal type

fpytloun commented 5 months ago

added retry function to bypass database locking. Would not be needed if frigate has option to use WAL journal type

Frigate does use the WAL journal type

Interesting. Even when I used PRAGMA journal_mode=WAL; prior to UPDATE statement I sometimes hit database locked error.

NickM-27 commented 5 months ago

Right, because there is still a write timeout that results in a locked error. If something is writing and something else waits for longer than that timeout it will have a locked error regardless of if it's using wal or not.

pgiansante commented 3 months ago

I tried the latest script above and I lose all access to the events via frigate app. Does this work with the 0.13.2?

fpytloun commented 3 months ago

I tried the latest script above and I lose all access to the events via frigate app. Does this work with the 0.13.2?

Yes, I am using it and it works fine.

jakubsuchybio commented 2 months ago

Hi guys, just want to report my findings with trying to use unionfs, then mergefs. I managed to get it to work. So basically this setup:

And go wrong it went constantly :(. I don't have stable connection between my 2 locations (the NFS mount is not located on local storage, but across internet via VPN) My network would sometime stall and disconnect. That the unionfs and mergefs can't survive. They have some politics for disconnects, but I couldn't get them to survive disconnects. Only way to restore the union/mergefs would be by restarting whole frigate vm :(

Having only archiving without being able to view the archive from frigate seems kinda pointless to me, that is why I was trying this solution.

One last solution that comes to my mind is to write custom script/program, which would basically do something similar to union/mergefs but with symlinks. So when the local storage gets full, it will move oldest to the NFS archive and replace it locally with a symlink. But that creates new problem. When the frigate tries to remove that by the retention, it will just remove the symlink and not the target of the link on the archive storage. So to get it to work I would have to keep track of the symlinks on some db (file, sql, or something) and when going through, if some symlink would get deleted by frigate, but it was still tracked in the db, the script/program would delete it also from the NFS archive..

As I'm thinking about it, it might actually work. I will try to do that sometime soon and report back the findings.

toxic0berliner commented 2 months ago

I think this feature request is quite enough justified, I believe it's only waiting for some dev to start tackling it. My opinion is archive will be slow and potentially subject to issues during transfer, but frigate could still let the user mount whatever archive storage as a separate path. Seems to me a more reasonable starting point than aiming at handling smb,NFS,S3,.... For me frigate should have 2 new settings: a keep time after which files are moved to the archive, and a path to said archive. It should handle the copy and delete files from the main storage only once the archive has been successfully written. It should log any failure but also simply retry the transfer at the next occasion. It should also handle actual deletion from the archive instead of the main storage when the usual keep policy expires. Storage page could also benefit from being modified to display main and archive storage space. I don't think it should be more complicated than that. (That said I also believe it's too much for me to try implementing so thanks in advance to anyone willing to tackle it)

johntdyer commented 2 months ago

One last solution that comes to my mind is to write custom script/program, which would basically do something similar to union/mergefs but with symlinks. So when the local storage gets full, it will move oldest to the NFS archive and replace it locally with a symlink. But that creates new problem. When the frigate tries to remove that by the retention, it will just remove the symlink and not the target of the link on the archive storage. So to get it to work I would have to keep track of the symlinks on some db (file, sql, or something) and when going through, if some symlink would get deleted by frigate, but it was still tracked in the db, the script/program would delete it also from the NFS archive..

@jakubsuchybio - Would rsync w/ the delete flag work ? Just thinking when Frigate deletes the linked file the rsync might recognize this ( untested ) and delete the linked file on the other storage... Just an untested idea.... pre-coffee :)

fpytloun commented 2 months ago

@jakubsuchybio Frigate recordings has path in database so you can have multiple paths instead of using mergefs which seems to be very fragile. So why don't you use my script? I am using this so I have 1 day on local storage and then move recordings to JuiceFS volume (using Wasabi S3 backend). I can view both local and remote recordings just fine and remote storage instability (eg Internet outage) does not cause any harm. https://github.com/blakeblackshear/frigate/issues/3673#issuecomment-2057730850

jakubsuchybio commented 2 months ago

@fpytloun

Juicefs, didn't hear of it, will look at it, thx for hint

yeah i wouldn't dare to modify db paths 😁

zzachattack2 commented 3 weeks ago

+1 for the feature request for tiered storage. I think the flexibility that Frigate already has on what to record, and how long to retain it, are great, but the missing part of where is a big shortcoming IMO -- especially considering the volume of data that can be involved here.

Just throwing out some ideas, but what I think would be ideal is if we could at least identify a "hot-storage" recording path and a "cold-storage" recording path, and each recording 'option' that currently allows us to specify a retention period, could also specify an archiving period before moving to cold storage.

Example use case: I want to keep a week's worth 24/7 recordings (~4TB), which means I pretty much need to store that on spinning platters. For recent footage of particular importance (e.g events with a person identified), which are substantially more likely to actually be viewed, it would be better to have it available on a faster storage medium. I also want to keep those events for much longer (say 30 days), but after a few days, the need for them on fast storage largely goes away (along with available space on the SSD), and they should be moved over to cold storage.

So a hypothetical configuration could look like:

record:
  enabled: True
  hot_path: /data/hot
  cold_path: /data/cold
  retain:
    mode: all
    days: 7
    archive: 0 #don't use hot-storage
  events:
    retain:
      mode: active_objects
      default: 
        days: 14
        archive: 1 #move to cold-storage after 1 day
      objects:
        person:
          days: 30
          archive: 3 #move to cold-storage after 3 days
javydekoning commented 3 weeks ago

+1 for the feature request for tiered storage. I think the flexibility that Frigate already has on what to record, and how long to retain it, are great, but the missing part of where is a big shortcoming IMO -- especially considering the volume of data that can be involved here.

Just throwing out some ideas, but what I think would be ideal is if we could at least identify a "hot-storage" recording path and a "cold-storage" recording path, and each recording 'option' that currently allows us to specify a retention period, could also specify an archiving period before moving to cold storage.

Example use case: I want to keep a week's worth 24/7 recordings (~4TB), which means I pretty much need to store that on spinning platters. For recent footage of particular importance (e.g events with a person identified), which are substantially more likely to actually be viewed, it would be better to have it available on a faster storage medium. I also want to keep those events for much longer (say 30 days), but after a few days, the need for them on fast storage largely goes away (along with available space on the SSD), and they should be moved over to cold storage.

So a hypothetical configuration could look like:

record:
  enabled: True
  hot_path: /data/hot
  cold_path: /data/cold
  retain:
    mode: all
    days: 7
    archive: 0 #don't use hot-storage
  events:
    retain:
      mode: active_objects
      default: 
        days: 14
        archive: 1 #move to cold-storage after 1 day
      objects:
        person:
          days: 30
          archive: 3 #move to cold-storage after 3 days

I like this and suggest splitting the datastore to it's own object given the (future) possibility for remote/cloud storage:

  datastore:
  - name: my-hot-volume
    type: hostPath
    config:
      path: /data/hot
  - name: my-cold-bucket
    type: s3
    config:
      bucket: s3://my-bucket
      prefix: /frigate
      storage_class: 'STANDARD'