search

ep1cman / unifi-protect-backup

Python tool to backup unifi event clips in realtime
MIT License
635 stars 31 forks source link

Retry events until completed #50

Closed jsumners closed 1 year ago

jsumners commented 2 years ago

Sometimes ISP connections are unstable. Such was the case for me on 2022-09-16. My connection was basically unusable for the whole day until I re-synchronized my "modem" with the ISP. During this time, at least one event was recorded and could not be uploaded to the remote store. The issue ends up looking like the attached screenshot.

I think this issue can be solved at the same time as #49. What I am thinking, is that a local sqlite or leveldb store can be used to keep track of the events received and successful uploads. When the script purges items from the remote storage, it can also remove records from this local tracking store (i.e. implement a sliding window of tracked events).

hp-elite-desk:0 0 0 tmux docker logs -f protect-backup 2022-09-18 11-45-31

ep1cman commented 2 years ago

Hi,

just wanted to say I’ve seen this and I understand your problem and will look into fixing it at some point. However, I just got a puppy so my spare time is somewhat limited in the near term

jsumners commented 2 years ago

That's cool. Thank you for the acknowledgement. I'd help if I were at all competent with Python (and also had any time 🤣).

ep1cman commented 1 year ago

Hi,

Thank you for your patience. This feature should now implemented in the latest update. The tool now ensures all events within the retention window are backed up, even if they occurred before it was first run.

The check runs periodically, so if some event failed to download, it should retry it with every check for missing events until it suceeds

jsumners commented 1 year ago

awwwwwwww-excited

Swallowtail23 commented 1 year ago

This is the only pip-installed program I use... so apologies if this is a noob question.

I'm running 0.7.4 at the moment, and when running: pip install --upgrade unifi-protect-backup ...I just get a lot of "Requirement already satisfied" and nothing upgrades.

What is the best way to upgrade please?

ep1cman commented 1 year ago

My apologies looks like the CI failed to upload the release to PyPI. Please check back in an hour or so and hopefully it will be resolved by then

Swallowtail23 commented 1 year ago

No worries, and thanks for your work on this, it's much appreciated.

Other than (when it's available) running the pip upgrade, is everything else the same, no gotchas with config etc., it works the same as before?

ep1cman commented 1 year ago

If you are using the pip version there should be no change in your use of the tool. The only potential issue is that the tool now creates a SQLite database that you need to make sure persists (mostly an issue with running the tool as a container)

jsumners commented 1 year ago

mostly an issue with running the tool as a container

Does it default to creating the database in the already mounted data volume?

ep1cman commented 1 year ago

mostly an issue with running the tool as a container

Does it default to creating the database in the already mounted data volume?

It defaults to /config/database/events.sqlite

Swallowtail23 commented 1 year ago

On my RH (7) server /config does not exist at all...

I notice: --sqlite_path. The Help text says "Path to the SQLite database to use/create" - is it the path to the database, i.e. it needs to be a full path including filename? Or is it a path to the containing folder (in which case can I suggest changing the text to something like "Path to the folder to be used for the SQLite database to use/create (folder needs to be persistent on reboot)").

Swallowtail23 commented 1 year ago

OK, upgraded, and tested with setting a --sqlite_path to a db file, and that looks like it is what it wanted, so all good there. It picked up some missed events on start and has uploaded those.

Thanks Sebastian - great upgrade.

Swallowtail23 commented 1 year ago

I did notice one peculiarity.

To upgrade I (accidentally) stopped the previous version in the middle of it backing up an event, and then upgraded and restarted (in debug to make sure the db was connecting OK). You can see the sequence here:

Dec  5 09:32:31 emp75 unifi_protect: 2022-12-05 09:32:31 [INFO]:unifi_protect_backup.unifi_protect_backup:#011Backing up event: 638d2dfc0139bf03e4034612
Dec  5 09:32:42 emp75 unifi_protect: 2022-12-05 09:32:42 [INFO]:unifi_protect_backup.unifi_protect_backup:#011Backed up successfully!
Dec  5 09:33:19 emp75 unifi_protect: 2022-12-05 09:33:19 [INFO]:unifi_protect_backup.unifi_protect_backup:#011Backing up event: 638d2e03031dbf03e403461b
Dec  5 09:35:00 emp75 unifi_protect: 2022-12-05 09:35:00 [#033[36;1m   DEBUG   #033[0m] unifi_protect_backup.unifi_protect_backup  :#011Config:
Dec  5 09:35:00 emp75 unifi_protect: 2022-12-05 09:35:00 [#033[36;1m   DEBUG   #033[0m] unifi_protect_backup.unifi_protect_backup  :#011  address='192.168.1.18'

The event that was backing up (638d2e03031dbf03e403461b) when the service was stopped has not been uploaded to OneDrive, and is not being picked up as 'missed'.

I've not tested to see if any subsequent restarts in the middle of an action result in the same miss.

ep1cman commented 1 year ago

On my RH (7) server /config does not exist at all...

That path was for inside the container sorry. If your are running it directly on the machine it defaults to events.sqlite in the current directory.

I notice: --sqlite_path. The Help text says "Path to the SQLite database to use/create" - is it the path to the database, i.e. it needs to be a full path including filename? Or is it a path to the containing folder (in which case can I suggest changing the text to something like "Path to the folder to be used for the SQLite database to use/create (folder needs to be persistent on reboot)").

It means the full path to the database file, not a directory

As for your above issue, I will open a new issue to track it and keep the discussion there rather than in this closed issue