kartoza / docker-pg-backup

A cron job that will back up databases running in a docker postgres container
GNU General Public License v2.0
452 stars 103 forks source link

clean_s3bucket: oldest files not removed #71

Closed mbl-35 closed 1 year ago

mbl-35 commented 1 year ago

Hi, I think I've find a bug in the cleanup process. Here are the traces

root@7d14762e73e7:/backup-scripts# /backup-scripts/backups.sh
Bucket 's3://vault-backups/' created
upload: '<stdin>' -> 's3://vault-backups/globals.sql'  [part 1 of -, 481B] [1 of 1]
 481 of 481   100% in    0s    35.24 KB/s  done
WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
upload: '/vault-backups/2023/March/PG_ts_vault.17-March-2023.dmp.gz' -> 's3://vault-backups/2023/March/PG_ts_vault.17-March-2023.dmp.gz'  [1 of 1]
 15364 of 15364   100% in    0s     2.16 MB/s  done
Done. Uploaded 15364 bytes in 1.0 seconds, 15.00 KB/s.
awk: line 1: syntax error at or near {
awk: line 1: syntax error at or near {
awk: line 1: syntax error at or near }
date: invalid date ‘+%s’
date: invalid date ‘-vault-backups’
awk: line 1: syntax error at or near {
awk: line 1: syntax error at or near {
awk: line 1: syntax error at or near }
date: invalid date ‘+%s’
date: invalid date ‘-vault-backups’

It sound that the function will not calculate proper dates:

# Cleanup S3 bucket
function clean_s3bucket() {
  S3_BUCKET=$1
  DEL_DAYS=$2
  s3cmd ls s3://${S3_BUCKET} --recursive | while read -r line; do
    createDate=$(echo $line | awk {'print ${S3_BUCKET}" "${DEL_DAYS}'})
    createDate=$(date -d"$createDate" +%s)
    olderThan=$(date -d"-${S3_BUCKET}" +%s)
    if [[ $createDate -lt $olderThan ]]; then
      fileName=$(echo $line | awk {'print $4'})
      echo $fileName
      if [[ $fileName != "" ]]; then
        s3cmd del "$fileName"
      fi
    fi
  done
}

should be change with:

# Cleanup S3 bucket
function clean_s3bucket() {
  S3_BUCKET=$1
  DEL_DAYS=$2
  s3cmd ls s3://${S3_BUCKET} --recursive | while read -r line; do
    createDate=$(echo $line | awk {'print $1'})
    createDate=$(date -d"$createDate" +%s)
    olderThan=$(date -d"${DEL_DAYS} ago" +%s)
    if [[ $createDate -lt $olderThan ]]; then
      fileName=$(echo $line | awk {'print $4'})
      echo $fileName
      if [[ $fileName != "" ]]; then
        s3cmd del "$fileName"
      fi
    fi
  done
}

Think it will be better :)

NyakudyaA commented 1 year ago

Thanks for picking it up, Can you do a PR with these changes