search

ep1cman / unifi-protect-backup

Python tool to backup unifi event clips in realtime
MIT License
633 stars 31 forks source link

Oversized events from Unifi Protect cause the docker container to crash #107

Closed kitleyn closed 11 months ago

kitleyn commented 11 months ago

Description

During my initial upload of historic events I noticed that two events were causing the container to run out of memory and crash. Logs showed the below entries. Both are Ring events from a G4 doorbell and seem to last an unusually long time (34.5hrs and 4.55hrs respectively). I assume this is a Unifi bug causing them to be this long but open to feedback. From some quick calculations, I estimate these files to be ~26.5GiB and ~3.5GiB respectively which is obviously far too big to load into the buffer and also exceeds both the memory allocations of my Docker container and the disk space of the host. How could these rogue events be excluded and/or alerted from unifi-protect-backup in the future through some file size limit or duration limit? Has anybody else seen these types of long running events from Unifi Protect API?

Many thanks

2023-10-20 16:28:16 [   INFO    ] unifi_protect_backup.downloader            : [6529241c02537003e400e3f2]  Downloading event: 6529241c02537003e400e3f2
2023-10-20 16:28:16 [   DEBUG   ] unifi_protect_backup.downloader            : [6529241c02537003e400e3f2]  Remaining Download Queue: 0
2023-10-20 16:28:16 [   DEBUG   ] unifi_protect_backup.downloader            : [6529241c02537003e400e3f2]  Video Download Buffer: 0.0B/512.0MiB
2023-10-20 16:28:16 [   DEBUG   ] unifi_protect_backup.downloader            : [6529241c02537003e400e3f2]    Camera: Front Door
2023-10-20 16:28:16 [   DEBUG   ] unifi_protect_backup.downloader            : [6529241c02537003e400e3f2]    Type: ring
2023-10-20 16:28:16 [   DEBUG   ] unifi_protect_backup.downloader            : [6529241c02537003e400e3f2]    Start: 2023-10-13T12-03-56 (1697195036.591)
2023-10-20 16:28:16 [   DEBUG   ] unifi_protect_backup.downloader            : [6529241c02537003e400e3f2]    End: 2023-10-14T22-25-16 (1697318716.897)
2023-10-20 16:28:16 [   DEBUG   ] unifi_protect_backup.downloader            : [6529241c02537003e400e3f2]    Duration: 123680.306s
2023-10-20 16:28:16 [   DEBUG   ] unifi_protect_backup.downloader            : [6529241c02537003e400e3f2]    Downloading video...
2023-10-20 20:08:47 [   DEBUG   ] unifi_protect_backup.downloader            : [652ac748032d7003e401043f]  Video Download Buffer: 22.9MiB/512.0MiB
2023-10-20 20:08:47 [   DEBUG   ] unifi_protect_backup.downloader            : [652ac748032d7003e401043f]    Camera: Front Door
2023-10-20 20:08:47 [   DEBUG   ] unifi_protect_backup.downloader            : [652ac748032d7003e401043f]    Type: ring
2023-10-20 20:08:47 [   DEBUG   ] unifi_protect_backup.downloader            : [652ac748032d7003e401043f]    Start: 2023-10-14T17-52-24 (1697302344.809)
2023-10-20 20:08:47 [   DEBUG   ] unifi_protect_backup.downloader            : [652ac748032d7003e401043f]    End: 2023-10-14T22-25-16 (1697318716.897)
2023-10-20 20:08:47 [   DEBUG   ] unifi_protect_backup.downloader            : [652ac748032d7003e401043f]    Duration: 16372.088s
2023-10-20 20:08:47 [   DEBUG   ] unifi_protect_backup.downloader            : [652ac748032d7003e401043f]    Downloading video...

What I Did

To force skip these files/events, I stopped the container, edited the sqlite DB to include the ID and associated data of the entries and then restarted. This prevented them from being uploaded but was a manual process.

ep1cman commented 11 months ago

This is a tricky problem to solve, something has clearly gone horribly wrong with unifi protect.

What I think I will do is as some sort of maximum event length setting that defaults to say 2h, anything longer will get ignored. If users want longer events they can change the setting.

ep1cman commented 11 months ago

Implemented a maximum length option in v0.10. Please comment back if you still have issues