search

eeshugerman / postgres-backup-s3

A handy Docker container to periodically backup PostgreSQL to S3
MIT License
485 stars 167 forks source link

Backups not being deleted #23

Closed RicardoViteriR closed 1 year ago

RicardoViteriR commented 2 years ago

Hello, I noticed that my backups are not being deleted from S3 - I am using DigitalOcean as it has the same API as Amazon S3. Wondering what may be the issue, my config is as follow:

      BACKUP_KEEP_DAYS: '1'
      POSTGRES_DATABASE: postgres
      POSTGRES_HOST: vpn.myhost.com
      POSTGRES_PASSWORD: SECRET
      POSTGRES_PORT: '5433'
      POSTGRES_USER: backups_user
      S3_ACCESS_KEY_ID: SECRET_ID
      S3_BUCKET: AWESOME
      S3_ENDPOINT: https://nyc3.digitaloceanspaces.com
      S3_PREFIX: backup-pg-production
      S3_REGION: nyc3
      S3_SECRET_ACCESS_KEY: /ACCESS
      SCHEDULE: '@every 0h15m0s'
eeshugerman commented 2 years ago

Hmm 🤔 is there anything in the container logs?

RicardoViteriR commented 2 years ago

Everything looks okay from the container log. 

2022/10/14 19:44:49 3327 cmd: /bin/sh backup.sh
2022/10/14 19:44:49 3327: Creating dump of postgres database from vpn.123.com...
2022/10/14 19:48:05 3327: Uploading dump to my-space
2022/10/14 19:48:18 3327: SQL backup uploaded successfully

Wondering if the expiration date is set per file or per bucket?

eeshugerman commented 1 year ago

Sorry I'm not sure 😕. I didn't implement the auto-removal feature and tbh barely understand the unholy shell incantations it uses. Sometimes I'm tempted to rewrite the whole thing in Python... But please report back if you find a solution!

eeshugerman commented 1 year ago

Oh, meant to ask, have you tried with BACKUP_KEEP_DAYS > 1? I wonder if it's some kind of edge case.

RicardoViteriR commented 1 year ago

@eeshugerman hi, I tested with BACKUP_KEEP_DAYS > 1 for a few days and it does not erase the file. Unfortunately, I have not looked into it and tinker how to make things work. Wondering if it has anything to do that I am using DigitalOcean as my hosting provider. They are supposed to be S3 compatible.

Will look into the source code to understand what the logic is to delete files.

eeshugerman commented 1 year ago

Well, thanks for trying it!

I use Digital Ocean too, though not with BACKUP_KEEP_DAYS. I just enabled it though, will report back with results in a week or so.

RicardoViteriR commented 1 year ago

Hi @eeshugerman wondering if you ever go to test this?

eeshugerman commented 1 year ago

Hi @RicardoViteriR, I just checked and BACKUP_KEEP_DAYS: 30 is working as expected for me on DigitalOcean spaces.

RicardoViteriR commented 1 year ago

Hi @eeshugerman, I'd like to apologize for opening this issue and not being careful of details. I mistakenly was using another container which has the same configuration options and confused it with yours. I just reran the docker container with your image and deletion worked like a charm.

eeshugerman commented 1 year ago

@RicardoViteriR No worries! Thanks for the update.