search

ep1cman / unifi-protect-backup

Python tool to backup unifi event clips in realtime
MIT License
576 stars 25 forks source link

Events which are larger than the buffer size never get abandoned #88

Closed Sticklyman1936 closed 1 year ago

Sticklyman1936 commented 1 year ago

Description

It appears that an event which is larger than the buffer size was generated. Every so often the tool is retrying the download and failing because the event is larger than the buffer. It seems that this event is not being abandoned after N attempts like the others, and instead keeps retrying. I don't know if it disappears at the end of the retention window or not as it has not been long enough to find out.

This event should either be abandoned after N attempts like some others are, or arguably immediately abandoned as it can never successfully download. (I presume the alternative would be to dynamically grow the buffer size with a default and a maximum, where maximum is only used if required. Abandon if larger than max. Just thinking out loud...).

What I Did

I am running with the latest docker image on github. The config is:

2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :  Config:
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    address='10.10.0.254'
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    port=443
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    username='protectsync'
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    password=REDACTED
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    verify_ssl=False
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    rclone_destination='Dropbox:/UnifiProtect'
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    retention='7d'
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    rclone_args='"--bwlimit=3000k"'
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    ignore_cameras=()
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    verbose=1
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    detection_types=['motion', 'person', 'vehicle', 'ring']
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    file_structure_format='{camera_name}/{event.start:%Y-%m-%d}/{event.end:%Y-%m-%dT%H-%M-%S} {detection_type}.mp4'
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    sqlite_path='/config/database/events.sqlite'
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    download_buffer_size=512.0MiB
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    purge_interval='1d'
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    apprise_notifiers=()
2023-04-27 19:04:57 [   DEBUG   ] unifi_protect_backup.unifi_protect_backup_core :    skip_missing=False

The error I am seeing is:

ERROR
2023-04-27 19:07:20 [   ERROR   ] unifi_protect_backup.downloader            : [6449950c014ac403e40003f8]  Unexpected exception occurred, abandoning event 6449950c014ac403e40003f8:
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/unifi_protect_backup/downloader.py", line 155, in start
    await self.upload_queue.put((event, video))
  File "/usr/lib/python3.10/site-packages/unifi_protect_backup/utils.py", line 413, in put
    raise ValueError(
ValueError: Item is larger (952.3MiB) than the size of the buffer (512.0MiB)
ERROR
2023-04-27 19:12:05 [   ERROR   ] unifi_protect_backup.downloader            : [6449950c014ac403e40003f8]  Unexpected exception occurred, abandoning event 6449950c014ac403e40003f8:
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/unifi_protect_backup/downloader.py", line 155, in start
    await self.upload_queue.put((event, video))
  File "/usr/lib/python3.10/site-packages/unifi_protect_backup/utils.py", line 413, in put
    raise ValueError(
ValueError: Item is larger (952.3MiB) than the size of the buffer (512.0MiB)
ERROR
2023-04-27 19:17:24 [   ERROR   ] unifi_protect_backup.downloader            : [6449950c014ac403e40003f8]  Unexpected exception occurred, abandoning event 6449950c014ac403e40003f8:
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/unifi_protect_backup/downloader.py", line 155, in start
    await self.upload_queue.put((event, video))
  File "/usr/lib/python3.10/site-packages/unifi_protect_backup/utils.py", line 413, in put
    raise ValueError(
ValueError: Item is larger (952.3MiB) than the size of the buffer (512.0MiB)
ep1cman commented 1 year ago

Should now be resolved in v0.9.1. Please re-open if it happens again.