search

ep1cman / unifi-protect-backup

Python tool to backup unifi event clips in realtime
MIT License
576 stars 25 forks source link

Errors when using --retention other than default #98

Closed grafpaper1987 closed 8 months ago

grafpaper1987 commented 11 months ago

Description

I'm trying to invoke the --retention option. I'd like to set retention to 90 days / 3 months. When I add this flag, unifi-protect-backup throws errors.

What I Did

I ran:

sudo unifi-protect-backup --address 'IP' --username 'NAME' --password 'PASS' --rclone-destination 'gdrive:Backup' --no-verify-ssl --detection-types 'person' --ignore-camera '636074f3026fc603e70003f6' --ignore-camera '636074f301dac603e70003f0' --ignore-camera '63dae0a302a2bc03e400fe8e' --ignore-camera '63e3f333033cbc03e4067dad' --ignore-camera '63851c3801392c03e4011115' --retention '3M'

I got the following:

2023-07-23 15:08:50 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : Starting... 2023-07-23 15:08:50 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : Checking rclone configuration... 2023-07-23 15:08:50 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : Connecting to Unifi Protect... 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : Found cameras: 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : - 636074f3026fc603e70003f6: Pool - Low 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : - 636074f301dac603e70003f0: Kitchen 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : - 63e3f333033cbc03e4067dad: Co-working Upstairs 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : - 63dae0a302a2bc03e400fe8e: Co-working Downstairs 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : - 636074f30225c603e70003f3: Bar 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : - 636074f30210c603e70003f2: Reception 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : - 636074f3023cc603e70003f4: Billiards Room 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : - 636074f301f8c603e70003f1: Beer Garden 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : - 636074f30197c603e70003ee: Front - Left 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : - 636074f301c0c603e70003ef: Front - Right 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : - 636074f30253c603e70003f5: Reception 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : - 63851c3801392c03e4011115: Pool - High 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.unifi_protect_backup_core : Starting Tasks... 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.downloader : Starting Downloader 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.uploader : Starting Uploader 2023-07-23 15:08:51 [ INFO ] unifi_protect_backup.missing_event_checker : Starting Missing Event Checker 2023-07-23 15:09:51 [ WARNING ] pyunifiprotect.api : Websocket connection not active, failing back to polling 2023-07-23 15:09:51 [ WARNING ] unifi_protect_backup.event_listener : Lost connection to Unifi Protect. 2023-07-23 15:09:51 [ WARNING ] unifi_protect_backup.event_listener : Attempting reconnect... 2023-07-23 15:09:51 [ ERROR ] unifi_protect_backup.missing_event_checker : Unexpected exception occurred during missing event check: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/unifi_protect_backup/missing_event_checker.py", line 131, in start wanted_events = await self._get_missing_events() File "/usr/local/lib/python3.10/dist-packages/unifi_protect_backup/missing_event_checker.py", line 59, in _get_missing_events unifi_events = await self._protect.get_events( File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 720, in get_events response = await self.get_events_raw( File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 690, in get_events_raw return await self.api_request_list("events", params=params) File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 345, in api_request_list data = await self.api_request( File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 311, in api_request data = await self.api_request_raw( File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 278, in api_request_raw response = await self.request(method, url, require_auth=require_auth, auto_close=False, **kwargs) File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 240, in request response = await req_context.aenter() # pylint: disable=unnecessary-dunder-call File "/usr/local/lib/python3.10/dist-packages/aiohttp/client.py", line 1141, in aenter self._resp = await self._coro File "/usr/local/lib/python3.10/dist-packages/aiohttp/client.py", line 400, in _request raise RuntimeError("Session is closed") RuntimeError: Session is closed 2023-07-23 15:10:46 [ ERROR ] unifi_protect_backup.event_listener : Unexpected exception occurred while trying to reconnect: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/unifi_protect_backup/event_listener.py", line 117, in _check_websocket_and_reconnect await self._protect.update(force=True) File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 600, in update self._bootstrap = await self.get_bootstrap() File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 763, in get_bootstrap data = await self.api_request_obj("bootstrap") File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 328, in api_request_obj data = await self.api_request( File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 311, in api_request data = await self.api_request_raw( File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 288, in api_request_raw raise BadRequest(msg % (url, response.status, reason)) pyunifiprotect.exceptions.BadRequest: Request failed: /proxy/protect/api/bootstrap - Status: 404 - Reason: Not Found 2023-07-23 15:10:56 [ WARNING ] unifi_protect_backup.event_listener : Attempting reconnect... 2023-07-23 15:10:56 [ ERROR ] unifi_protect_backup.event_listener : Unexpected exception occurred while trying to reconnect: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/unifi_protect_backup/event_listener.py", line 117, in _check_websocket_and_reconnect await self._protect.update(force=True) File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 600, in update self._bootstrap = await self.get_bootstrap() File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 763, in get_bootstrap data = await self.api_request_obj("bootstrap") File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 328, in api_request_obj data = await self.api_request( File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 311, in api_request data = await self.api_request_raw( File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 288, in api_request_raw raise BadRequest(msg % (url, response.status, reason)) pyunifiprotect.exceptions.BadRequest: Request failed: /proxy/protect/api/bootstrap - Status: 404 - Reason: Not Found 2023-07-23 15:11:06 [ WARNING ] unifi_protect_backup.event_listener : Attempting reconnect... 2023-07-23 15:11:07 [ ERROR ] unifi_protect_backup.event_listener : Unexpected exception occurred while trying to reconnect: Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/unifi_protect_backup/event_listener.py", line 117, in _check_websocket_and_reconnect await self._protect.update(force=True) File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 600, in update self._bootstrap = await self.get_bootstrap() File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 763, in get_bootstrap data = await self.api_request_obj("bootstrap") File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 328, in api_request_obj data = await self.api_request( File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 311, in api_request data = await self.api_request_raw( File "/usr/local/lib/python3.10/dist-packages/pyunifiprotect/api.py", line 288, in api_request_raw raise BadRequest(msg % (url, response.status, reason)) pyunifiprotect.exceptions.BadRequest: Request failed: /proxy/protect/api/bootstrap - Status: 404 - Reason: Not Found 2023-07-23 15:11:17 [ WARNING ] unifi_protect_backup.event_listener : Attempting reconnect... ^C2023-07-23 15:11:54 [ WARNING ] aiorun : Stopping the loop

ep1cman commented 11 months ago

The errors seem u related to the retention period. You seem to be immediately losing connection to UniFi protect and unable to re-establish a connection.

The retention period is handled by the upload side of the program so I don't see how changing that could have any effect in the ability to connect to UniFi protect

grafpaper1987 commented 11 months ago

The application works 100% as intended when I do not include the retention flag/option and has been running rock solid for the last week.

Please let me know if there are specific logs that I can provide that would shed more light on this issue? Thank you!

ep1cman commented 11 months ago

I think this might be related to #95. I am starting to think requesting too much event data from NVRs is crashing them.

Could you try with various sizes of retention to see if occurs at a certain point?

grafpaper1987 commented 11 months ago

Okay.... had a chance to play around a bit. I can set the retention up to '12d'. Anything greater than that and I get the errors pasted in my original post.

ep1cman commented 8 months ago

If my guess that this is caused by querying the server for too many events, then this should be resolved now with v0.9.5