How to compress data directory after backup in barman?

[wasiualhasib]

Barman is working well but there is an issue of compressing the data directory. Look at the below details of the backup. It backed up the whole database as expected but the of each backup is 26GB excluding WAL size. I want to compress that 26GB size how can I do it after backup using Barman? Is there any way to do it?

[barman@DB-1]# barman list-backup pg 
pg 20240608T000302 - Sat Jun  8 00:13:08 2024 - Size: 26.2 GiB - WAL Size: 4.6 GiB
pg 20240607T000302 - Fri Jun  7 00:12:33 2024 - Size: 26.2 GiB - WAL Size: 15.3 GiB
pg 20240606T033234 - Thu Jun  6 03:33:51 2024 - Size: 26.1 GiB - WAL Size: 6.3 GiB

[barman@DB-1 ~]$ cd /data/barman/pg/
[barman@DB-1 pg]$ ll
total 8
drwxr-x---  5 barman barman   75 Jun  8 03:34 base
drwxr-x---  2 barman barman    6 Jun  1 05:00 errors
-rw-r-----  1 barman barman   63 Jun  1 05:01 identity.json
drwxr-x---  2 barman barman    6 Jun  3 19:56 incoming
drwxr-x---  2 barman barman   46 Jun  8 12:36 streaming
drwxr-x--- 73 barman barman 4096 Jun  8 12:32 walls

[barman@DB-1 pg]$ cd base/
[barman@DB-1 base]$ ll
total 0
drwxr-x--- 3 barman barman 37 Jun  6 03:33 20240606T033234
drwxr-xr-x 3 barman barman 37 Jun  7 00:18 20240607T000302
drwxr-xr-x 3 barman barman 37 Jun  8 00:20 20240608T000302

[barman@DB-1 base]$ cd 20240606T033234/
[barman@DB-1 20240606T033234]$ ll
total 8
-rw-r-----  1 barman barman 1272 Jun  6 03:33 backup.info
drwx------ 20 barman barman 4096 Jun  6 03:33 data

Here this size is basically due to data directory of backup ID: 20240606T033234

[barman@PBVL-TEST-K8S-DB-1 20240606T033234]$ du -sch *
4.0K    backup.info
27G     data
27G     total

Is there anyone can suggest me how to compress data directory in this case? Because if actual database size is 1TB backup and backuped database size is also 1TB then I need to have at least 2-3TB free space to get multiple database backup.

I know there is a pre and post script configuration but I don't understand how to do it for each backup. Is here anyone can give me an idea about it? Because in the near future database size will be 100TB to 200TB in that case it is difficult to manage using dump backup.

[martinmarques]

Which version of Postgres and Barman? That plays a big role in compression options.

I suggest you start by reading this: http://rhaas.blogspot.com/2022/05/parallel-server-side-backup-compression.html And after that, go into the compression features in PG15 and on: https://www.postgresql.org/docs/15/app-pgbasebackup.html In particular, check the --compress options

[martinmarques]

And BTW, you will need a recent version of Barman that supports those compression values, and will need to use backup_method = postgres

[wasiualhasib]

Yes I got it but, if you set backup_method=postgres that means you are not using incremental backup. If I use backup_method=postgres does it allow PITR?

[andremagui]

PITR is not tied and related to incremental backups. PITR is related to continuous archiving of the WAL files + existence of a parent backup. So, if you have created a base backup (full), set archive_mode and archive_command in your postgresql.conf file and the archiving is happening, you are able to do PITR. Incremental backups help to reduce storage and speeds up recovery when replaying WALs, but you can do PITR without it.

[wasiualhasib]

Yes you are right. I have tested PITR, it was working without incremental backup. But I found incremental backup only work when I set at postgres: archive_mode=on and archive_command is set and at barman configuration: backup_method=rsync and reuse_method=link. But if I remove archive_command and set wal streaming wal through replication slot in that case incremental backup not working. In that time it works as like full backup. Did you notice that?

Compression is only supported at backup_method=postgres For rsync or incremental backup mybe I have to do it with manual scripts.

I fall in love with barman backup tools when I understood yes it is possible in different ways.

[martinmarques]

@wasiualhasib I believe you would get more answers to your questions in the barman google here

But I will answer the questions from your last reply: incremental backups and compression are currently incompatible options. You can compress or have deduplication via rsync link mode. But you can't do incremental backups if the backups are compressed. Hopefully, we will get this enhancement in Postgres 18 when supporting incremental backups in tar format.