search

ep1cman / unifi-protect-backup

Python tool to backup unifi event clips in realtime
MIT License
576 stars 25 forks source link

Error on Dev version and Latest version #151

Closed biofects closed 3 weeks ago

biofects commented 1 month ago

Description

On my OS I have a mount from a storage device to store my events. /Protect This is world readable and writable I have tested the latest and the dev version you have released bot throw different errors

What I Did

Dev version Docker Compose file

services:
    ep1cman:
        environment:
          - UFP_USERNAME=**************
          - UFP_PASSWORD=**************
          - UFP_ADDRESS=**************
          - UFP_SSL_VERIFY=true
          - RCLONE_RETENTION=60d
        volumes:
          - '/Protect/cameras:/data'
          - '/home/user/user-scripts/unifi-protect/database:/config/database'
          - './resolv.conf:/etc/resolv.conf'
        image: ghcr.io/ep1cman/unifi-protect-backup:dev
        restart: unless-stopped
        pull_policy: always

Error

p1cman-1  | 2024-05-30 21:21:40 [   INFO    ] unifi_protect_backup.downloader            : [663350a701d3fe03e4012950]  Downloading event: 663350a701d3fe03e4012950
ep1cman-1  | 2024-05-30 21:21:40 [   DEBUG   ] unifi_protect_backup.downloader            : [663350a701d3fe03e4012950]  Remaining Download Queue: 0
ep1cman-1  | 2024-05-30 21:21:40 [   DEBUG   ] unifi_protect_backup.downloader            : [663350a701d3fe03e4012950]  Video Download Buffer: 0.0B/512.0MiB
ep1cman-1  | 2024-05-30 21:21:40 [   DEBUG   ] unifi_protect_backup.downloader            : [663350a701d3fe03e4012950]    Camera: Front Yard Left
ep1cman-1  | 2024-05-30 21:21:40 [   DEBUG   ] unifi_protect_backup.downloader            : [663350a701d3fe03e4012950]    Type: motion
ep1cman-1  | 2024-05-30 21:21:40 [   DEBUG   ] unifi_protect_backup.downloader            : [663350a701d3fe03e4012950]    Start: 2024-05-02T03-36-52 (1714639012.408)
ep1cman-1  | 2024-05-30 21:21:40 [   DEBUG   ] unifi_protect_backup.downloader            : [663350a701d3fe03e4012950]    End: 2024-05-02T03-37-05 (1714639025.908)
ep1cman-1  | 2024-05-30 21:21:40 [   DEBUG   ] unifi_protect_backup.downloader            : [663350a701d3fe03e4012950]    Duration: 13.5s
ep1cman-1  | 2024-05-30 21:21:40 [   DEBUG   ] unifi_protect_backup.downloader            : [663350a701d3fe03e4012950]    Downloading video...
ep1cman-1  | 2024-05-30 21:21:41 [  WARNING  ] unifi_protect_backup.downloader            : [663350a701d3fe03e4012950]  Event failed download attempt 1
ep1cman-1  | Traceback (most recent call last):
ep1cman-1  |   File "/usr/lib/python3.12/site-packages/unifi_protect_backup/downloader.py", line 144, in start
ep1cman-1  |     video = await self._download(event)
ep1cman-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ep1cman-1  |   File "/usr/lib/python3.12/site-packages/unifi_protect_backup/downloader.py", line 184, in _download
ep1cman-1  |     video = await self._protect.get_camera_video(event.camera_id, event.start, event.end)
ep1cman-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ep1cman-1  |   File "/usr/lib/python3.12/site-packages/pyunifiprotect/api.py", line 1206, in get_camera_video
ep1cman-1  |     return await self.api_request_raw(
ep1cman-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ep1cman-1  |   File "/usr/lib/python3.12/site-packages/pyunifiprotect/api.py", line 249, in api_request_raw
ep1cman-1  |     await self._os.verify_response(url, response)
ep1cman-1  |   File "/usr/lib/python3.12/site-packages/pyunifiprotect/session.py", line 331, in verify_response
ep1cman-1  |     await self.clear_auth()
ep1cman-1  |   File "/usr/lib/python3.12/site-packages/pyunifiprotect/session.py", line 424, in clear_auth
ep1cman-1  |     await self.session_cache.delete_session(session_hash)
ep1cman-1  |   File "/usr/lib/python3.12/site-packages/pyunifiprotect/session.py", line 143, in delete_session
ep1cman-1  |     cache = await self.read_session_cache()
ep1cman-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ep1cman-1  |   File "/usr/lib/python3.12/site-packages/pyunifiprotect/session.py", line 77, in read_session_cache
ep1cman-1  |     async with aiofiles.open(self.session_cache, "rb") as f:
ep1cman-1  |   File "/usr/lib/python3.12/site-packages/aiofiles/base.py", line 98, in __aenter__
ep1cman-1  |     self._obj = await self._coro
ep1cman-1  |                 ^^^^^^^^^^^^^^^^
ep1cman-1  |   File "/usr/lib/python3.12/site-packages/aiofiles/threadpool/__init__.py", line 94, in _open
ep1cman-1  |     f = yield from loop.run_in_executor(executor, cb)
ep1cman-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ep1cman-1  |   File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
ep1cman-1  |     result = self.fn(*self.args, **self.kwargs)
ep1cman-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ep1cman-1  | PermissionError: [Errno 13] Permission denied: '/root/.cache/ufp/sessions.json'
biofects commented 1 month ago

I managed to get it to download some or all the videos, waiting on the sync to finish. I still get the error Permission denied

rknightion commented 1 month ago

I'm getting the same thing. I'm wondering if it's an issue inside the container itself

biofects commented 1 month ago

So digging in a bit more I see the sub process of the app is styring to run as abc user and write to a root file. Im looking on how I cna modify that

rknightion commented 1 month ago

I did try manually creating the folder/file and giving it 777 and even tried mounting it into the container from the local filesystem yet it still throws the error my side

biofects commented 1 month ago

I did try manually creating the folder/file and giving it 777 and even tried mounting it into the container from the local filesystem yet it still throws the error my side

So did I.. I am looking at the code not to see what is causing this

ep1cman commented 1 month ago

Looks like pyunifiprotect uses a new file it didn't before. Sorry I'm away at https://www.emfcamp.org/ and won't be able to look into it till next week. If someone does solve it though I will accept a pull request

biofects commented 1 month ago

Looks like pyunifiprotect uses a new file it didn't before. Sorry I'm away at https://www.emfcamp.org/ and won't be able to look into it till next week. If someone does solve it though I will accept a pull request

Have a great time. That looks awesome

biofects commented 1 month ago

Im going to try a simple hack in the docker_root run file and see if that works 



# Create directory with permissions for user 'abc' (read, write, execute)
mkdir -p /root/.cache/ufp/ && chmod 700 /root/.cache/ufp/

# Create file with permissions for user 'abc' (read, write)
touch /root/.cache/ufp/sessions.json && chmod 600 /root/.cache/ufp/sessions.json

# Export Rclone config path
export RCLONE_CONFIG=/config/rclone/rclone.conf

# Check for verbosity flag
echo $VERBOSITY

# Set verbosity argument based on flag presence
[[ -n "$VERBOSITY" ]] && export VERBOSITY_ARG=-$VERBOSITY || export VERbosity_ARG=""

# Execute the command with s6-setuidgid
exec s6-setuidgid abc unifi-protect-backup ${VERBOSITY_ARG}```
biofects commented 1 month ago

Well need to see why I get make docker failure

docker buildx build . --platform linux/amd64,linux/arm64 -t ghcr.io/ep1cman/unifi-protect-backup --push unknown flag: --platform

rknightion commented 1 month ago

The makefile I think was for an older version of buildx.

Anyways, I got it working with a nasty hack of setting an env var XDG_CACHE_HOME to /tmp and it's working now. Loads of 401s but no more permission denied :)

biofects commented 1 month ago

hehe Yea Im doing a docker build now

biofects commented 1 month ago

hmm ok so far no luck this is super strange I added 

RUN chown abc:abc /root/.cache/ufp

and it creates the file, it is there in the container but still same error

Im going to keep pokiing and figure this out (I hope)

hydazz commented 1 month ago

hmm ok so far no luck this is super strange I added

RUN chown abc:abc /root/.cache/ufp

and it creates the file, it is there in the container but still same error

Im going to keep pokiing and figure this out (I hope)

Try move the chown to here if you haven't already. chowns dont work properly within the dockerfile, as the GUID and PUID for abc are changed

biofects commented 1 month ago

hmm ok so far no luck this is super strange I added

RUN chown abc:abc /root/.cache/ufp

and it creates the file, it is there in the container but still same error Im going to keep pokiing and figure this out (I hope)

Try move the chown to here if you haven't already. chowns dont work properly within the dockerfile, as the GUID and PUID for abc are changed

Thanks I moved it all to the 30- config, Still getting the same error
PermissionError: [Errno 13] Permission denied: '/root/.cache/ufp/sessions.json

The file is created and owned properly

-rw-r--r-- 1 abc abc 0 Jun  2 17:17 /root/.cache/ufp/sessions.json

This is a strange one but we shall see

hydazz commented 1 month ago

hmm ok so far no luck this is super strange I added

RUN chown abc:abc /root/.cache/ufp

and it creates the file, it is there in the container but still same error Im going to keep pokiing and figure this out (I hope)

Try move the chown to here if you haven't already. chowns dont work properly within the dockerfile, as the GUID and PUID for abc are changed

Thanks I moved it all to the 30- config, Still getting the same error PermissionError: [Errno 13] Permission denied: '/root/.cache/ufp/sessions.json

The file is created and owned properly

-rw-r--r-- 1 abc abc 0 Jun  2 17:17 /root/.cache/ufp/sessions.json

This is a strange one but we shall see

export the home to /config, just above where the service is executed export HOME=/config. ufp should then create the cache in /config, then there should be no permission issues

biofects commented 4 weeks ago

Ok I have reverted everything and I added the export to the run file and looks like the errors are gone. I still get some errors 

  File "/usr/lib/python3.12/site-packages/unifi_protect_backup/downloader.py", line 144, in start
    video = await self._download(event)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/unifi_protect_backup/downloader.py", line 184, in _download
    video = await self._protect.get_camera_video(event.camera_id, event.start, event.end)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pyunifiprotect/api.py", line 1206, in get_camera_video
    return await self.api_request_raw(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pyunifiprotect/api.py", line 249, in api_request_raw
    await self._os.verify_response(url, response)
  File "/usr/lib/python3.12/site-packages/pyunifiprotect/session.py", line 332, in verify_response
    raise NotAuthorized(msg % (url, response.status, reason))

Also when I restarted the test build a bunch of 401's were returned

ep1cman commented 4 weeks ago

All of these are fixing the symptom rather than the issue.

It seems pyunifiprotect now caches session details in a file, the path to which is either provided when instantiating ProtectApiClient (done in unifi_protect_backup_core.py) or it defaults to using a library called platformdirs.

The correct solution would be to figure out why platform dirs is thinking the user is root when it isnt executed as root. My guess is s6-setuidgid is not setting the uid/gid properly and somehow the root user is getting picked up by platformdirs.

I do not have the time to look into this further right now but will at some point later in the week, or at the latest over the weekend.

ep1cman commented 4 weeks ago

According to: https://github.com/platformdirs/platformdirs/blob/2f7aecf2a1eaa95d0e60960a5cc1fc0651c4c90c/src/platformdirs/unix.py#L25

It looks like this library is based on a really big assumption, it is run in a desktop environment, not a command line environment. Should be easy enough to set the necessary env variable XDG_DATA_HOME to something abc has R/W access to, probably /config

ep1cman commented 3 weeks ago

This should be fixed in the latest dev container.