backup to S3 not working

[sushilbansal]

Hi, I tried using the S3 bucket; backup container is not running. I tested the AWS credentials and bucket using some other means and bucket is definitely connecting and working.

version: '3'
volumes:
  postgres-data:
services:
  node-docker:
    container_name: node_container
    build: .
    ports:
      - 3000:4000

  postgres:
    container_name: postgres_container
    image: postgis/postgis
    restart: always
    environment:
      - POSTGRES_DB=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=password
    ports:
      - '5432:5432'
    volumes:
      - postgres-data:/var/lib/postgresql/data

  dbbackups:
    container_name: dbbackups_container
    image: kartoza/pg-backup:${POSTGRES_MAJOR_VERSION:-14}-${POSTGIS_MAJOR_VERSION:-3}.${POSTGIS_MINOR_RELEASE:-1}
    environment:
      - DUMPPREFIX=backup
      - POSTGRES_HOST=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASS=password
      - POSTGRES_PORT=5432
      - CRON_SCHEDULE="*/5 * * * *"
      - STORAGE_BACKEND="S3"
      - ACCESS_KEY_ID=MY_AWS_KEY_ID
      - SECRET_ACCESS_KEY=MY_AWS_SECRET_KEY
      - DEFAULT_REGION=eu-west-2
      - BUCKET=MY_AWS_BUCKET
      - SSL_SECURE=True
    restart: on-failure
    depends_on:
      postgres:
        condition: service_healthy

[NyakudyaA]

Hi, I tried using the S3 bucket; backup container is not running. I tested the AWS credentials and bucket using some other means and bucket is definitely connecting and working.

version: '3'

volumes:

  postgres-data:

services:

  node-docker:

    container_name: node_container

    build: .

    ports:

      - 3000:4000

  postgres:

    container_name: postgres_container

    image: postgis/postgis

    restart: always

    environment:

      - POSTGRES_DB=postgres

      - POSTGRES_USER=postgres

      - POSTGRES_PASSWORD=password

    ports:

      - '5432:5432'

    volumes:

      - postgres-data:/var/lib/postgresql/data

  dbbackups:

    container_name: dbbackups_container

    image: kartoza/pg-backup:${POSTGRES_MAJOR_VERSION:-14}-${POSTGIS_MAJOR_VERSION:-3}.${POSTGIS_MINOR_RELEASE:-1}

    environment:

      - DUMPPREFIX=backup

      - POSTGRES_HOST=postgres

      - POSTGRES_USER=postgres

      - POSTGRES_PASS=password

      - POSTGRES_PORT=5432

      - CRON_SCHEDULE="*/5 * * * *"

      - STORAGE_BACKEND="S3"

      - ACCESS_KEY_ID=MY_AWS_KEY_ID

      - SECRET_ACCESS_KEY=MY_AWS_SECRET_KEY

      - DEFAULT_REGION=eu-west-2

      - BUCKET=MY_AWS_BUCKET

      - SSL_SECURE=True

    restart: on-failure

    depends_on:

      postgres:

        condition: service_healthy

It seems you are missing a couple of env variables needed for the configuration of s3cfg Please check the example docker-compose

[mbl-35]

Using the latest version 14-3.2, the PATH environment variable is not properly set and /usr/local/bin is missing for cron tasks to be able to access to the /usr/local/bin/s2cmd command. As a workaround, I've redifined the container entrypoint to /backup-scripts/start-with-path.sh with following content:

#!/bin/bash
# adds path for cron tasks to have s3cmd in path
awk '1;/export DBLIST=/{print "export PATH='$PATH'"}' /backup-scripts/start.sh > tmp && mv tmp /backup-scripts/start.sh
chmod u+x /backup-scripts/start.sh
/backup-scripts/start.sh

This script only append a line in the generated /backup-scripts/pgenv.sh file with the 'export PATH'. Mount that file in docker compose to /backup-scripts/start-with-path.sh, and all will be sucessfull.

As a merge request, we could append that line in the scripts/start.sh file OR modify all scripts with the full path to the s3cmd.

[NyakudyaA]

@mbl-35 Can you make a PR and I will test this. I am also going to add this to tests so that we can be sure this is covered. The cron runs in a subshell so we need to make it available. How about just adding

export PATH=\"${PATH}\"

in https://github.com/kartoza/docker-pg-backup/blob/master/scripts/start.sh

[mbl-35]

@NyakudyaA : yes, it was the idea.. just putting export PATH=\"${PATH}\" in the script/starts.sh while generating the /backup-scripts/pgenv.sh file That's what doing my workaround ... and its working (local tested with minio).. This is less secured for a cron job (normaly don't have /usr/local/bin in $PATH to protect system), but less impact on existing pg-backup code.