search

ep1cman / unifi-protect-backup

Python tool to backup unifi event clips in realtime
MIT License
577 stars 25 forks source link

Retention window is being ignored. No purging happening at all. #79

Closed cyberpower678 closed 9 months ago

cyberpower678 commented 1 year ago

Description

Retention period is not being followed. It's set to 2 months (2M), but it's got footage from 3 months ago and it's still accumulating. The log suggests its set to purge every day.

What I Did

Restart the container.
ep1cman commented 1 year ago

Are you still having this issue?

cyberpower678 commented 1 year ago

Yes

On Mar 23, 2023, at 19:50, Sebastian Goscik @.***> wrote:

Are you still having this issue?

— Reply to this email directly, view it on GitHub https://github.com/ep1cman/unifi-protect-backup/issues/79#issuecomment-1482058592, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALIL262XYLRWIQJJ27I7ITW5TOVTANCNFSM6AAAAAAU2RV4B4. You are receiving this because you authored the thread.

ep1cman commented 1 year ago

Sorry for the delay, I didn't see that you had responded. Could you let me know which storage provider you are using? A log of the application running with verbosity set to vv would be helpful too.

brettacarr commented 1 year ago

I have the same issue. I'm on Unifi Protect Backup version: 0.9.0. running in Raspberry Pi 4.

It's working great other than that, thank you :)

ep1cman commented 1 year ago

@brettacarr Can you share your logs with verbosity set to -vv ?

brettacarr commented 1 year ago

Hey @ep1cman , I've looked a little deeper. It seems the purge command was running but failing due to having the wrong remote name in the DB (I'd renamed the remote but records in the DB had the old name). I've updated to DB records to have the correct remote name and it looks like it successfully purged videos from 2023-01-28 (90 days ago). It only seems to have purged files from that day and not days prior to that. Is that expected behaviour?

ep1cman commented 1 year ago

The way the code works is as follows:

1) It selects all the entries in the events table that are older than the retention period: https://github.com/ep1cman/unifi-protect-backup/blob/1b38cb3db3bd051539f61188f79bc8498d581b83/unifi_protect_backup/purge.py#L58-L61

2) It then selects all the entries in the backup table with matching event IDs (it is setup like this to in the future support multiple backup locations): https://github.com/ep1cman/unifi-protect-backup/blob/1b38cb3db3bd051539f61188f79bc8498d581b83/unifi_protect_backup/purge.py#L68

3) it then attempts to delete the file

4) It deletes the entry in the events table: https://github.com/ep1cman/unifi-protect-backup/blob/1b38cb3db3bd051539f61188f79bc8498d581b83/unifi_protect_backup/purge.py#L76-L77

So what I suspect has happened: It tried to delete the file, but failed for whatever reason, then deleted the event entry and is no longer trying to delete that event leaving the files still there.

I see a few flaws with this approach: 1) Deletes event entries regardless of if there are still backup entries, it assumes all the deletes succeeded (the database is setup to auto delete backup entries when their corresponding event is deleted https://github.com/ep1cman/unifi-protect-backup/blob/1b38cb3db3bd051539f61188f79bc8498d581b83/unifi_protect_backup/unifi_protect_backup_core.py#L40) 2) Because of 1. it will never attempt to retry deleting a backup

@brettacarr would you be willing to share your database with me so I can have a look at whats going on? You can email it to github-issue@goscik.com

I am going to try and re-work this code to make this process more robust

brettacarr commented 1 year ago

Sorry for the delay. I've emailed through the DB.

It's been purging each day now since the 28th Jan, but all the earlier files are not being picked up.

Sounds like you might need to only commit the DB transaction after you know the file has been successfully removed.