barman-cloud-backup can't take the backup from remote postgres server

[genral73]

Hi. I have tested barman with remote server and wal streaming replication of my postgresql server. I have also tested barman cloud to push on a s3 bucket in a dedicated server but something went wrong. the file pg_control is not found when I did the backup

barman-cloud-backup -e AES256 -j --immediate-checkpoint -J 4 -h *** -U -P barman-cloud s3:// 2021-03-03 20:00:22,135 [346] ERROR: Backup failed uploading data ([Errno 2] No such file or directory: '/var/lib/postgresql/11/main/global/pg_control') Exception ignored in: <bound method _Stream.del of <tarfile._Stream object at 0x7f2aa2ace550>> Traceback (most recent call last): File "/usr/lib/python3.6/tarfile.py", line 419, in del self.close() File "/usr/lib/python3.6/tarfile.py", line 467, in close self.fileobj.write(self.buf) File "/usr/lib/python3/dist-packages/barman/cloud.py", line 195, in write self.buffer = self._buffer() File "/usr/lib/python3.6/tempfile.py", line 690, in NamedTemporaryFile (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type) File "/usr/lib/python3.6/tempfile.py", line 401, in _mkstemp_inner fd = _os.open(file, flags, 0o600) FileNotFoundError: [Errno 2] No such file or directory: '/tmp/barman-cloud-backup-400haj93/barman-upload-_ff81kov.part'

Does it mean I can't use barman cloud on a remote dedicated server to backup postgres data with barman cloud ? If yes, why ? I mean, it better to make the dedicated server working instead of the postgres server to avoid any over workloads ?

Second question is: barman-cloud doesnt support database param, that means I have to connect with a postgresql account with already created database. In my case, I have an account replication which doesnt have database so is there a way to connect with an account and specify postgres database for example like barman conninfo ?

Thanks

[didiermichel]

Hello @genral73, barman cloud can be used in a dedicated server in conjunction with barman. To do so you need to set post_backup_script command in barman (something like this: post_backup_script=barman-cloud-backup s3://your_bucket/path ${BARMAN_SERVER}). The command will be activated each time you run the barman backup on the server.

[genral73]

Hi @didiermichel As you explained, I tested it. As a result, the post backup script didn't upload the New Backup to s3 bucket.

Please find the config file.

[db-1]
description = "Backups for db-1"
ssh_command = "ssh -q postgres@192.168.56.61"
conninfo = "host=192.168.56.61 port=5432 dbname=postgres user=barman "
streaming_conninfo = "host=192.168.56.61 port=5432 dbname=postgres user=streaming_barman "
streaming_archiver = on
slot_name = "backup_db_1"
active = true
post_backup_script=barman-cloud-backup -P minio --endpoint-url http://minio.localhost:9000 s3://barman/test ${BARMAN_SERVER}

Output:

[barman@db-5 ~]$ barman backup db-1 
Starting backup using rsync-concurrent method for server db-1 in /data/barman/db-1/base/20221019T113823
Backup start at LSN: 4/F0000028 (0000000100000004000000F0, 00000028)
This is the first backup for server db-1
WAL segments preceding the current backup have been found:
    0000000100000004000000EE from server db-1 has been removed
Starting backup copy via rsync/SSH for 20221019T113823 (12 jobs)
Copy done (time: 1 minute)
This is the first backup for server db-1
WAL segments preceding the current backup have been found:
    0000000100000004000000EF from server db-1 has been removed
Asking PostgreSQL server to finalize the backup.
Backup size: 5.7 GiB. Actual size on disk: 5.7 GiB (-0.00% deduplication ratio).
Backup end at LSN: 4/F0000138 (0000000100000004000000F0, 00000138)
Backup completed (start time: 2022-10-19 11:38:24.052634, elapsed time: 1 minute, 2 seconds)
Processing xlog segments from streaming for db-1
    0000000100000004000000F0
    0000000100000004000000F1

[barman@db-5 ~]$ barman list-backup db-1 
db-1 20221019T113823 - Wed Oct 19 08:39:24 2022 - Size: 5.7 GiB - WAL Size: 16.0 KiB

[barman@db-5 ~]$ aws s3 --profile minio --endpoint-url http://minio.localhost:9000 ls  s3://barman/test
[barman@db-5 ~]$ 
[barman@db-5 ~]$ 

[mikewallace1979]

Unfortunately Barman does not provide much information about hook scripts in the command output.

Try setting log_level = debug in the Barman config and then check the contents of /var/log/barman/barman.log - there should be some lines with the string barman.hooks which log the command and its exit status. There should be some more useful output just above that line - in the case of a permissions error, for example, you would see something like this:

2022-10-21 15:48:56,469 [977] barman.hooks DEBUG: Attempt to run backup_script: barman-cloud-backup --endpoint=http://minio:9000 s3://test-buckt/test ${BARMAN_SERVER}
2022-10-21 15:48:56,469 [977] barman.command_wrappers DEBUG: Command: 'barman-cloud-backup --endpoint=http://minio:9000 s3://test-buckt/test ${BARMAN_SERVER}'
2022-10-21 15:48:56,821 [977] Command WARNING: 2022-10-21 15:48:56,821 [1053] ERROR: Barman cloud backup exception: An error occurred (403) when calling the HeadBucket operation: Forbidden
2022-10-21 15:48:56,858 [977] barman.command_wrappers DEBUG: Command return code: 4
2022-10-21 15:48:56,858 [977] barman.command_wrappers DEBUG: Command stdout: 
2022-10-21 15:48:56,859 [977] barman.command_wrappers DEBUG: Command stderr: 2022-10-21 15:48:56,821 [1053] ERROR: Barman cloud backup exception: An error occurred (403) when calling the HeadBucket operation: Forbidden

2022-10-21 15:48:56,859 [977] barman.hooks WARNING: barman-cloud-backup --endpoint=http://minio:9000 s3://test-buckt/test ${BARMAN_SERVER} returned 4
Output details:
2022-10-21 15:48:56,821 [1053] ERROR: Barman cloud backup exception: An error occurred (403) when calling the HeadBucket operation: Forbidden