search

ep1cman / unifi-protect-backup

Python tool to backup unifi event clips in realtime
MIT License
633 stars 31 forks source link

How to handle rclone rate limit errors #120

Open axsuul opened 9 months ago

axsuul commented 9 months ago

Description

I'm getting these errors which seems to be related to rclone being rate limited

ERROR
2023-12-18 06:14:09 [   ERROR   ] unifi_protect_backup.utils                 :  stderr:
    2023/12/18 06:13:37 DEBUG : Setting --config "/config/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/config/rclone/rclone.conf"
    2023/12/18 06:13:37 DEBUG : rclone: Version "v1.64.2-DEV" starting with parameters ["rclone" "rcat" "-vv" "remote:backup-bucket/Front Yard/2023-12-17/2023-12-17T22-13-14 motion.mp4"]
    2023/12/18 06:13:37 DEBUG : Creating backend with remote "remote:backup-bucket/Front Yard/2023-12-17/"
    2023/12/18 06:13:37 DEBUG : Using config file from "/config/rclone/rclone.conf"
    2023/12/18 06:13:38 DEBUG : Couldn't decode error response: EOF
    2023/12/18 06:13:38 DEBUG : fs cache: renaming cache item "remote:backup-bucket/Front Yard/2023-12-17/" to be canonical "remote:backup-bucket/Front Yard/2023-12-17"
    2023/12/18 06:13:38 DEBUG : 2023-12-17T22-13-14 motion.mp4: File has 15790479 bytes, which makes only one chunk. Using direct upload.
    2023/12/18 06:14:09 DEBUG : 2023-12-17T22-13-14 motion.mp4: Clearing upload URL because of error: incident id 96bbc0f6cf18-c51bdb438ab74a77 (500 internal_error)
    2023/12/18 06:14:09 DEBUG : pacer: low level retry 1/1 (error incident id 96bbc0f6cf18-c51bdb438ab74a77 (500 internal_error))
    2023/12/18 06:14:09 DEBUG : pacer: Rate limited, increasing sleep to 20ms
    2023/12/18 06:14:09 ERROR : 2023-12-17T22-13-14 motion.mp4: Post request rcat error: incident id 96bbc0f6cf18-c51bdb438ab74a77 (500 internal_error)
    2023/12/18 06:14:09 DEBUG : 6 go routines active
    2023/12/18 06:14:09 Failed to rcat with 2 errors: last error was: incident id 96bbc0f6cf18-c51bdb438ab74a77 (500 internal_error)
ERROR
2023-12-18 06:14:09 [   ERROR   ] unifi_protect_backup.utils                 :  stdout:
ERROR
2023-12-18 06:14:09 [   ERROR   ] unifi_protect_backup.uploader              : [657fe2fd020ca003e40889cb]   Failed to upload file: 'remote:backup-bucket/Front Yard/2023-12-17/2023-12-17T22-13-14 motion.mp4'
ERROR
2023-12-18 06:14:09 [   ERROR   ] unifi_protect_backup.utils                 :  Failed to run: 'rclone rcat -vv  "remote:backup-bucket/Front Yard/2023-12-17/2023-12-17T22-13-14 motion.mp4"

Is there something I can do on my end to address this or force a retry without notifying about the error? Thanks!

adidalal commented 5 months ago

What backend are you using here? You can additionally add some timer or rate limit flags to rclone to avoid the issue