search

EnterpriseDB / barman

Barman - Backup and Recovery Manager for PostgreSQL
https://www.pgbarman.org/
GNU General Public License v3.0
2.07k stars 191 forks source link

Add information to `barman show-backup` about incremental backups #964

Closed andremagui closed 1 month ago

andremagui commented 2 months ago

Now that Postgres 17 have the ability to take incremental backups with pg_basebackup, there is a need to add new information to the show-backup command.

This PR will:

  1. Add fields to the get_backup_ext_info method from Server so we have extra information to be reused in the outputs.
  2. Modify output.py to accommodate the new information that came with the introduction of incremental backups.
  3. Fix tests broken by this implementation
  4. Add unit tests according to use cases.

Note: The code in the output.py have to deal with two inputs: the result of the get_backup_ext_info method call (which is a dictionary with the extra fields mentioned in 1.) or backup_info.to_dict() (which is for cloud-show-backup). The latter will not have the new fields mentioned in 1. This is also the case for --format json for barman-cloud-backup-show. It relies on the backup_info.to_json() method. The code implemented is a little repetitive so we do not break the barman-cloud-backup-show command.

References: BAR-208 & BAR-214

barthisrael commented 2 months ago

I have performed some tests to check how things look like.

I have these backups, both from Postgres 17. The custom names of the backup should give you an idea of what they are 😄 :

$ barman list-backup all
pg17 20240801T201448 'pg_incremental' - I - Thu Aug  1 18:14:53 2024 - Size: 365.9 MiB - WAL Size: 0 B - WAITING_FOR_WALS
pg17 20240801T201244 'pg_full' - F - Thu Aug  1 18:12:47 2024 - Size: 251.5 MiB - WAL Size: 656.0 MiB
support 20240801T201405 'rsync_incremental' - R - Thu Aug  1 18:14:14 2024 - Size: 581.3 MiB - WAL Size: 32.0 MiB
support 20240801T201228 'rsync_full' - R - Thu Aug  1 18:12:36 2024 - Size: 235.2 MiB - WAL Size: 672.0 MiB

This is how show-backup of a rsync full looks like:

$ barman show-backup support rsync_full
Backup 20240801T201228:
  Backup Name            : rsync_full
  Server Name            : support
  System Id              : 7382227563826631291
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data
  Estimated Cluster Size : 219.0 MiB

  Server information:
    Checksums            : off
    WAL summarizer       : on

  Tablespaces:
    custom_ts_1          : /var/lib/pgsql/17/tablespaces/custom_ts_1 (oid: 24678)
    custom_ts_2          : /var/lib/pgsql/17/tablespaces/custom_ts_2 (oid: 24686)

  Base backup information:
    Backup Method        : rsync-concurrent
    Backup Size          : 219.2 MiB (235.2 MiB with WALs)
    WAL Size             : 16.0 MiB
    Resource savings     : 0 B (0.00%)
    Timeline             : 1
    Begin WAL            : 000000010000000600000052
    End WAL              : 000000010000000600000052
    WAL number           : 1
    Begin time           : 2024-08-01 18:12:28.516824+00:00
    End time             : 2024-08-01 18:12:36.132905+00:00
    Copy time            : 5 seconds + 1 second startup
    Estimated throughput : 38.6 MiB/s
    Begin Offset         : 40
    End Offset           : 344
    Begin LSN            : 6/52000028
    End LSN              : 6/52000158

  WAL information:
    No of files          : 42
    Disk usage           : 672.0 MiB
    WAL rate             : 1573.45/hour
    Last available       : 00000001000000060000007C

  Catalog information:
    Retention Policy     : not enforced
    Previous Backup      : - (this is the oldest base backup)
    Next Backup          : 20240801T201405

This is the diff compared to master branch:

8c8,12
<   Checksums              : off
---
>   Estimated Cluster Size : 219.0 MiB
>
>   Server information:
>     Checksums            : off
>     WAL summarizer       : on
15,16c19,22
<     Disk usage           : 219.2 MiB (235.2 MiB with WALs)
<     Incremental size     : 219.2 MiB (-0.00%)
---
>     Backup Method        : rsync-concurrent
>     Backup Size          : 219.2 MiB (235.2 MiB with WALs)
>     WAL Size             : 16.0 MiB
>     Resource savings     : 0 B (0.00%)

This is how rsync incremental looks like:

$ barman show-backup support rsync_incremental
Backup 20240801T201405:
  Backup Name            : rsync_incremental
  Server Name            : support
  System Id              : 7382227563826631291
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data
  Estimated Cluster Size : 565.1 MiB

  Server information:
    Checksums            : off
    WAL summarizer       : on

  Tablespaces:
    custom_ts_1          : /var/lib/pgsql/17/tablespaces/custom_ts_1 (oid: 24678)
    custom_ts_2          : /var/lib/pgsql/17/tablespaces/custom_ts_2 (oid: 24686)

  Base backup information:
    Backup Method        : rsync-concurrent
    Backup Size          : 347.5 MiB (363.5 MiB with WALs)
    WAL Size             : 16.0 MiB
    Resource savings     : 217.7 MiB (38.53%)
    Timeline             : 1
    Begin WAL            : 00000001000000060000007C
    End WAL              : 00000001000000060000007C
    WAL number           : 1
    Begin time           : 2024-08-01 18:14:05.743646+00:00
    End time             : 2024-08-01 18:14:14.509817+00:00
    Copy time            : 6 seconds + 1 second startup
    Estimated throughput : 57.8 MiB/s
    Begin Offset         : 37752
    End Offset           : 1029312
    Begin LSN            : 6/7C009378
    End LSN              : 6/7C0FB4C0

  WAL information:
    No of files          : 2
    Disk usage           : 32.0 MiB
    WAL rate             : 289.85/hour
    Last available       : 00000001000000060000007E

  Catalog information:
    Retention Policy     : not enforced
    Previous Backup      : 20240801T201228
    Next Backup          : - (this is the latest base backup)

This is the diff compared to master branch:

8c8,12
<   Checksums              : off
---
>   Estimated Cluster Size : 565.1 MiB
>
>   Server information:
>     Checksums            : off
>     WAL summarizer       : on
15,16c19,22
<     Disk usage           : 565.3 MiB (581.3 MiB with WALs)
<     Incremental size     : 347.5 MiB (-38.53%)
---
>     Backup Method        : rsync-concurrent
>     Backup Size          : 347.5 MiB (363.5 MiB with WALs)
>     WAL Size             : 16.0 MiB
>     Resource savings     : 217.7 MiB (38.53%)

This is how pg full looks like:

$ barman show-backup pg17 pg_full
Backup 20240801T201244:
  Backup Name            : pg_full
  Server Name            : pg17
  System Id              : 7382227563826631291
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data
  Estimated Cluster Size : 219.0 MiB

  Server information:
    Checksums            : off
    WAL summarizer       : on

  Tablespaces:
    custom_ts_1          : /var/lib/pgsql/17/tablespaces/custom_ts_1 (oid: 24678)
    custom_ts_2          : /var/lib/pgsql/17/tablespaces/custom_ts_2 (oid: 24686)

  Base backup information:
    Backup Method        : postgres
    Backup Type          : full
    Children Backup Id(s): ['20240801T201448']
    Backup Size          : 219.5 MiB (251.5 MiB with WALs)
    WAL Size             : 32.0 MiB
    Timeline             : 1
    Begin WAL            : 000000010000000600000054
    End WAL              : 000000010000000600000055
    WAL number           : 2
    Begin time           : 2024-08-01 18:12:44.372221+00:00
    End time             : 2024-08-01 18:12:47.489739+00:00
    Copy time            : 3 seconds
    Estimated throughput : 70.8 MiB/s
    Begin Offset         : 40
    End Offset           : 0
    Begin LSN            : 6/54000028
    End LSN              : 6/55000000

  WAL information:
    No of files          : 41
    Disk usage           : 656.0 MiB
    WAL rate             : 1227.44/hour
    Last available       : 00000001000000060000007E

  Catalog information:
    Retention Policy     : VALID
    Previous Backup      : - (this is the oldest base backup)
    Next Backup          : 20240801T201448

This is the diff compared to master branch:

8c8,12
<   Checksums              : off
---
>   Estimated Cluster Size : 219.0 MiB
>
>   Server information:
>     Checksums            : off
>     WAL summarizer       : on
15,16c19,23
<     Disk usage           : 219.5 MiB (251.5 MiB with WALs)
<     Incremental size     : 219.5 MiB (-0.00%)
---
>     Backup Method        : postgres
>     Backup Type          : full
>     Children Backup Id(s): ['20240801T201448']
>     Backup Size          : 219.5 MiB (251.5 MiB with WALs)
>     WAL Size             : 32.0 MiB

This is how pg incremental looks like:

$ barman show-backup pg17 pg_incremental
Backup 20240801T201448:
  Backup Name            : pg_incremental
  Server Name            : pg17
  System Id              : 7382227563826631291
  Status                 : WAITING_FOR_WALS
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data
  Estimated Cluster Size : 565.1 MiB

  Server information:
    Checksums            : off
    WAL summarizer       : on

  Tablespaces:
    custom_ts_1          : /var/lib/pgsql/17/tablespaces/custom_ts_1 (oid: 24678)
    custom_ts_2          : /var/lib/pgsql/17/tablespaces/custom_ts_2 (oid: 24686)

  Base backup information:
    Backup Method        : postgres
    Backup Type          : incremental
    Root Backup Id       : 20240801T201244
    Parent Backup Id     : 20240801T201244
    Backup chain size    : 2
    Backup Size          : 349.9 MiB (365.9 MiB with WALs)
    WAL Size             : 16.0 MiB
    Resource savings     : 215.3 MiB (38.09%)
    Timeline             : 1
    Begin WAL            : 00000001000000060000007E
    End WAL              : 00000001000000060000007F
    WAL number           : 1
    Begin time           : 2024-08-01 18:14:48.599869+00:00
    End time             : 2024-08-01 18:14:53.684562+00:00
    Copy time            : 5 seconds
    Estimated throughput : 69.2 MiB/s
    Begin Offset         : 40
    End Offset           : 0
    Begin LSN            : 6/7E000028
    End LSN              : 6/7F000000

  WAL information:
    No of files          : 0
    Disk usage           : 0 B
    Last available       : 00000001000000060000007E

  Catalog information:
    Retention Policy     : VALID
    Previous Backup      : 20240801T201244
    Next Backup          : - (this is the latest base backup)

This is the diff compared to master branch`:

8c8,12
<   Checksums              : off
---
>   Estimated Cluster Size : 565.1 MiB
>
>   Server information:
>     Checksums            : off
>     WAL summarizer       : on
15,16c19,26
<     Disk usage           : 349.9 MiB (365.9 MiB with WALs)
<     Incremental size     : 349.9 MiB (-0.00%)
---
>     Backup Method        : postgres
>     Backup Type          : incremental
>     Root Backup Id       : 20240801T201244
>     Parent Backup Id     : 20240801T201244
>     Backup chain size    : 2
>     Backup Size          : 349.9 MiB (365.9 MiB with WALs)
>     WAL Size             : 16.0 MiB
>     Resource savings     : 215.3 MiB (38.09%)
barthisrael commented 2 months ago

I'm putting the new "diff"s between master and this dev branch:

8c8,12
<   Checksums              : off
---
>   Estimated Cluster Size : 219.0 MiB
>
>   Server information:
>     Checksums            : off
>     WAL summarizer       : on
15,16c19,22
<     Disk usage           : 219.5 MiB (251.5 MiB with WALs)
<     Incremental size     : 219.5 MiB (-0.00%)
---
>     Backup Method        : postgres
>     Backup Type          : full
>     Backup Size          : 219.5 MiB (251.5 MiB with WALs)
>     WAL Size             : 32.0 MiB
39a46
>     Children Backup(s)   : 20240801T201448
6,7c6,11
<       "analysis_time": "less than one second",
<       "analysis_time_seconds": 0,
---
>       "backup_method": "postgres",
>       "backup_size": "219.5 MiB",
>       "backup_size_bytes": 230120784,
>       "backup_size_with_wals": "251.5 MiB",
>       "backup_size_with_wals_bytes": 263675216,
>       "backup_type": "full",
12a17,19
>       "children_backup_ids": [
>         "20240801T201448"
>       ],
15,18d21
<       "disk_usage": "219.5 MiB",
<       "disk_usage_bytes": 230120784,
<       "disk_usage_with_wals": "251.5 MiB",
<       "disk_usage_with_wals_bytes": 263675216,
24,27c27,29
<       "incremental_size": "219.5 MiB",
<       "incremental_size_bytes": 230120784,
<       "incremental_size_ratio": "-0.00%",
<       "number_of_workers": 1,
---
>       "resource_savings": "-478.6 KiB",
>       "resource_savings_bytes": -490061,
>       "resource_savings_percentage": "-0.21%",
29,30c31,35
<       "throughput_bytes": 74267896.02659073,
<       "timeline": 1
---
>       "throughput_bytes": 74267896,
>       "timeline": 1,
>       "wal_num": 2,
>       "wal_size": "32.0 MiB",
>       "wal_size_bytes": 33554432
37c42
<     "data_checksums": "off",
---
>     "cluster_size": 229630723,
39a45,48
>     "server_information": {
>       "data_checksums": "off",
>       "summarize_wal": "on"
>     },
40a50
>     "system_id": "7382227563826631291",
8c8,12
<   Checksums              : off
---
>   Estimated Cluster Size : 565.1 MiB
>
>   Server information:
>     Checksums            : off
>     WAL summarizer       : on
15,16c19,23
<     Disk usage           : 349.9 MiB (381.9 MiB with WALs)
<     Incremental size     : 349.9 MiB (-0.00%)
---
>     Backup Method        : postgres
>     Backup Type          : incremental
>     Backup Size          : 349.9 MiB (381.9 MiB with WALs)
>     WAL Size             : 32.0 MiB
>     Resource savings     : 215.3 MiB (38.09%)
39a47,50
>
>     Root Backup          : 20240801T201244
>     Parent Backup        : 20240801T201244
>     Backup chain size    : 2
6,7c6,11
<       "analysis_time": "less than one second",
<       "analysis_time_seconds": 0,
---
>       "backup_method": "postgres",
>       "backup_size": "349.9 MiB",
>       "backup_size_bytes": 366853563,
>       "backup_size_with_wals": "381.9 MiB",
>       "backup_size_with_wals_bytes": 400407995,
>       "backup_type": "incremental",
12a17,18
>       "chain_size": 2,
>       "children_backup_ids": null,
15,18d20
<       "disk_usage": "349.9 MiB",
<       "disk_usage_bytes": 366853563,
<       "disk_usage_with_wals": "381.9 MiB",
<       "disk_usage_with_wals_bytes": 400407995,
24,27c26,30
<       "incremental_size": "349.9 MiB",
<       "incremental_size_bytes": 366853563,
<       "incremental_size_ratio": "-0.00%",
<       "number_of_workers": 1,
---
>       "parent_backup_id": "20240801T201244",
>       "resource_savings": "215.3 MiB",
>       "resource_savings_bytes": 225707336,
>       "resource_savings_percentage": "38.09%",
>       "root_backup_id": "20240801T201244",
29,30c32,36
<       "throughput_bytes": 72526317.69362025,
<       "timeline": 1
---
>       "throughput_bytes": 72526317,
>       "timeline": 1,
>       "wal_num": 2,
>       "wal_size": "32.0 MiB",
>       "wal_size_bytes": 33554432
37c43
<     "data_checksums": "off",
---
>     "cluster_size": 592560899,
39a46,49
>     "server_information": {
>       "data_checksums": "off",
>       "summarize_wal": "on"
>     },
40a51
>     "system_id": "7382227563826631291",
8c8,12
<   Checksums              : off
---
>   Estimated Cluster Size : 219.0 MiB
>
>   Server information:
>     Checksums            : off
>     WAL summarizer       : on
15,16c19,22
<     Disk usage           : 219.2 MiB (235.2 MiB with WALs)
<     Incremental size     : 219.2 MiB (-0.00%)
---
>     Backup Method        : rsync-concurrent
>     Backup Size          : 219.2 MiB (235.2 MiB with WALs)
>     WAL Size             : 16.0 MiB
>     Resource savings     : 0 B (0.00%)
7a8,13
>       "backup_method": "rsync-concurrent",
>       "backup_size": "219.2 MiB",
>       "backup_size_bytes": 229797198,
>       "backup_size_with_wals": "235.2 MiB",
>       "backup_size_with_wals_bytes": 246574414,
>       "backup_type": "rsync",
15,18d20
<       "disk_usage": "219.2 MiB",
<       "disk_usage_bytes": 229797198,
<       "disk_usage_with_wals": "235.2 MiB",
<       "disk_usage_with_wals_bytes": 246574414,
24,27c26,28
<       "incremental_size": "219.2 MiB",
<       "incremental_size_bytes": 229797198,
<       "incremental_size_ratio": "-0.00%",
<       "number_of_workers": 1,
---
>       "resource_savings": "0 B",
>       "resource_savings_bytes": 0,
>       "resource_savings_percentage": "0.00%",
29,30c30,34
<       "throughput_bytes": 40525304.28737553,
<       "timeline": 1
---
>       "throughput_bytes": 40525304,
>       "timeline": 1,
>       "wal_num": 1,
>       "wal_size": "16.0 MiB",
>       "wal_size_bytes": 16777216
37c41
<     "data_checksums": "off",
---
>     "cluster_size": 229630723,
39a44,47
>     "server_information": {
>       "data_checksums": "off",
>       "summarize_wal": "on"
>     },
40a49
>     "system_id": "7382227563826631291",
8c8,12
<   Checksums              : off
---
>   Estimated Cluster Size : 565.1 MiB
>
>   Server information:
>     Checksums            : off
>     WAL summarizer       : on
15,16c19,22
<     Disk usage           : 565.3 MiB (581.3 MiB with WALs)
<     Incremental size     : 347.5 MiB (-38.53%)
---
>     Backup Method        : rsync-concurrent
>     Backup Size          : 347.5 MiB (363.5 MiB with WALs)
>     WAL Size             : 16.0 MiB
>     Resource savings     : 217.7 MiB (38.53%)
7a8,13
>       "backup_method": "rsync-concurrent",
>       "backup_size": "347.5 MiB",
>       "backup_size_bytes": 364332189,
>       "backup_size_with_wals": "363.5 MiB",
>       "backup_size_with_wals_bytes": 381109405,
>       "backup_type": "rsync",
15,18d20
<       "disk_usage": "565.3 MiB",
<       "disk_usage_bytes": 592727384,
<       "disk_usage_with_wals": "581.3 MiB",
<       "disk_usage_with_wals_bytes": 609504600,
24,27c26,28
<       "incremental_size": "347.5 MiB",
<       "incremental_size_bytes": 364332189,
<       "incremental_size_ratio": "-38.53%",
<       "number_of_workers": 1,
---
>       "resource_savings": "217.7 MiB",
>       "resource_savings_bytes": 228318416,
>       "resource_savings_percentage": "38.53%",
29,30c30,34
<       "throughput_bytes": 60653674.80849083,
<       "timeline": 1
---
>       "throughput_bytes": 60653674,
>       "timeline": 1,
>       "wal_num": 1,
>       "wal_size": "16.0 MiB",
>       "wal_size_bytes": 16777216
37c41
<     "data_checksums": "off",
---
>     "cluster_size": 592528131,
39a44,47
>     "server_information": {
>       "data_checksums": "off",
>       "summarize_wal": "on"
>     },
40a49
>     "system_id": "7382227563826631291",

I'm attaching the files to this reply, so we can later sit and discuss about this. pg_full_console_dev.txt pg_full_console_master.txt pg_full_json_dev.txt pg_full_json_master.txt pg_incremental_console_dev.txt pg_incremental_console_master.txt pg_incremental_json_dev.txt pg_incremental_json_master.txt rsync_full_console_dev.txt rsync_full_console_master.txt rsync_full_json_dev.txt rsync_full_json_master.txt rsync_incremental_console_dev.txt rsync_incremental_console_master.txt rsync_incremental_json_dev.txt rsync_incremental_json_master.txt

barthisrael commented 1 month ago

This PR needs to be rebased on top of master.

gcalacoci commented 1 month ago

This PR needs to be rebased on top of master.

Agreed. Also needs some squashing work while rebasing

barthisrael commented 1 month ago

I'll start throwing a set of comments here with the outcomes that I can see with the current implementation.

barthisrael commented 1 month ago

I took a backup with pg_basebackup using Barman 3.10.1.

Backup 20240808T191728:
  Server Name            : pg17
  System Id              : 7382227563826631291
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data

  Base backup information:
    Disk usage           : 25.3 MiB (57.3 MiB with WALs)
    Incremental size     : 25.3 MiB (-0.00%)
    Timeline             : 1
    Begin WAL            : 0000000100000007000000F2
    End WAL              : 0000000100000007000000F3
    WAL number           : 2
    Begin time           : 2024-08-08 17:17:28.190827+00:00
    End time             : 2024-08-08 17:17:33.493005+00:00
    Copy time            : 5 seconds
    Estimated throughput : 4.8 MiB/s
    Begin Offset         : 40
    End Offset           : 0
    Begin LSN            : 7/F2000028
    End LSN              : 7/F3000000

  WAL information:
    No of files          : 22
    Disk usage           : 352.0 MiB
    WAL rate             : 1223.59/hour
    Last available       : 000000010000000800000009

  Catalog information:
    Retention Policy     : VALID
    Previous Backup      : - (this is the oldest base backup)
    Next Backup          : - (this is the latest base backup)
Backup 20240808T191728:
  Server Name            : pg17
  System Id              : 7382227563826631291
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data

  Base backup information:
    Backup Method        : postgres
    Backup Type          : full
    Backup Size          : 25.3 MiB (57.3 MiB with WALs)
    WAL Size             : 32.0 MiB
    Timeline             : 1
    Begin WAL            : 0000000100000007000000F2
    End WAL              : 0000000100000007000000F3
    WAL number           : 2
    Begin time           : 2024-08-08 17:17:28.190827+00:00
    End time             : 2024-08-08 17:17:33.493005+00:00
    Copy time            : 5 seconds
    Estimated throughput : 4.8 MiB/s
    Begin Offset         : 40
    End Offset           : 0
    Begin LSN            : 7/F2000028
    End LSN              : 7/F3000000

  WAL information:
    No of files          : 22
    Disk usage           : 352.0 MiB
    WAL rate             : 1223.59/hour
    Last available       : 000000010000000800000009

  Catalog information:
    Retention Policy     : VALID
    Previous Backup      : - (this is the oldest base backup)
    Next Backup          : - (this is the latest base backup)
9,10c9,12
<     Disk usage           : 25.3 MiB (57.3 MiB with WALs)
<     Incremental size     : 25.3 MiB (-0.00%)
---
>     Backup Method        : postgres
>     Backup Type          : full
>     Backup Size          : 25.3 MiB (57.3 MiB with WALs)
>     WAL Size             : 32.0 MiB

This looks fine.

{
  "pg17": {
    "backup_id": "20240808T191728",
    "base_backup_information": {
      "analysis_time": "less than one second",
      "analysis_time_seconds": 0,
      "begin_lsn": "7/F2000028",
      "begin_offset": 40,
      "begin_time": "2024-08-08 17:17:28.190827+00:00",
      "begin_time_timestamp": "1723137448",
      "begin_wal": "0000000100000007000000F2",
      "copy_time": "5 seconds",
      "copy_time_seconds": 5.270819,
      "disk_usage": "25.3 MiB",
      "disk_usage_bytes": 26536648,
      "disk_usage_with_wals": "57.3 MiB",
      "disk_usage_with_wals_bytes": 60091080,
      "end_lsn": "7/F3000000",
      "end_offset": 0,
      "end_time": "2024-08-08 17:17:33.493005+00:00",
      "end_time_timestamp": "1723137453",
      "end_wal": "0000000100000007000000F3",
      "incremental_size": "25.3 MiB",
      "incremental_size_bytes": 26536648,
      "incremental_size_ratio": "-0.00%",
      "number_of_workers": 1,
      "throughput": "4.8 MiB/s",
      "throughput_bytes": 5034634.655449181,
      "timeline": 1
    },
    "catalog_information": {
      "next_backup": "- (this is the latest base backup)",
      "previous_backup": "- (this is the oldest base backup)",
      "retention_policy": "VALID"
    },
    "pgdata_directory": "/var/lib/pgsql/17/data",
    "postgresql_version": 170000,
    "status": "DONE",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "352.0 MiB",
      "disk_usage_bytes": 369098752,
      "last_available": "000000010000000800000009",
      "no_of_files": 22,
      "timelines": [],
      "wal_rate": "1223.59/hour",
      "wal_rate_per_second": 0.339887317436287
    }
  }
}
{
  "_EXCEPTION": [
    "float() argument must be a string or a number, not 'NoneType'\nSee log file for more details."
  ],
  "pg17": {
    "backup_id": "20240808T191728",
    "server_information": {
      "data_checksums": null,
      "summarize_wal": null
    },
    "status": "DONE",
    "system_id": "7382227563826631291"
  }
}

This breaks because of this:

See log file for more details.
Traceback (most recent call last):
  File "/home/vagrant/Repositories/edb/barman/barman/cli.py", line 2429, in main
    args.func(args)
  File "/home/vagrant/Repositories/edb/barman/barman/cli.py", line 1400, in show_backup
    server.show_backup(backup_info)
  File "/home/vagrant/Repositories/edb/barman/barman/server.py", line 3128, in show_backup
    output.result("show_backup", backup_ext_info)
  File "/home/vagrant/Repositories/edb/barman/barman/output.py", line 329, in result
    _dispatch(_writer, "result", command, *args, **kwargs)
  File "/home/vagrant/Repositories/edb/barman/barman/output.py", line 198, in _dispatch
    return handler(*args, **kwargs)
  File "/home/vagrant/Repositories/edb/barman/barman/output.py", line 1614, in result_show_backup
    cluster_size=pretty_size(data["cluster_size"]),
  File "/home/vagrant/Repositories/edb/barman/barman/utils.py", line 178, in pretty_size
    size = float(size)
TypeError: float() argument must be a string or a number, not 'NoneType'

If I work around that just to check how it would look like:

{
  "pg17": {
    "backup_id": "20240808T191728",
    "base_backup_information": {
      "backup_method": "postgres",
      "backup_size": "25.3 MiB",
      "backup_size_bytes": 26536648,
      "backup_size_with_wals": "57.3 MiB",
      "backup_size_with_wals_bytes": 60091080,
      "begin_lsn": "7/F2000028",
      "begin_offset": 40,
      "begin_time": "2024-08-08 17:17:28.190827+00:00",
      "begin_time_timestamp": "1723137448",
      "begin_wal": "0000000100000007000000F2",
      "copy_time": "5 seconds",
      "copy_time_seconds": 5.270819,
      "end_lsn": "7/F3000000",
      "end_offset": 0,
      "end_time": "2024-08-08 17:17:33.493005+00:00",
      "end_time_timestamp": "1723137453",
      "end_wal": "0000000100000007000000F3",
      "throughput": "4.8 MiB/s",
      "throughput_bytes": 5034634,
      "timeline": 1,
      "wal_num": 2,
      "wal_size": "32.0 MiB",
      "wal_size_bytes": 33554432
    },
    "catalog_information": {
      "backup_type": "full",
      "children_backup_ids": null,
      "next_backup": "- (this is the latest base backup)",
      "previous_backup": "- (this is the oldest base backup)",
      "retention_policy": "VALID"
    },
    "pgdata_directory": "/var/lib/pgsql/17/data",
    "postgresql_version": 170000,
    "server_information": {
      "data_checksums": null,
      "summarize_wal": null
    },
    "status": "DONE",
    "system_id": "7382227563826631291",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "352.0 MiB",
      "disk_usage_bytes": 369098752,
      "last_available": "000000010000000800000009",
      "no_of_files": 22,
      "timelines": [],
      "wal_rate": "1223.59/hour",
      "wal_rate_per_second": 0.339887317436287
    }
  }
}

And the diff would be:

5,6c5,9
<       "analysis_time": "less than one second",
<       "analysis_time_seconds": 0,
---
>       "backup_method": "postgres",
>       "backup_size": "25.3 MiB",
>       "backup_size_bytes": 26536648,
>       "backup_size_with_wals": "57.3 MiB",
>       "backup_size_with_wals_bytes": 60091080,
14,17d16
<       "disk_usage": "25.3 MiB",
<       "disk_usage_bytes": 26536648,
<       "disk_usage_with_wals": "57.3 MiB",
<       "disk_usage_with_wals_bytes": 60091080,
23,26d21
<       "incremental_size": "25.3 MiB",
<       "incremental_size_bytes": 26536648,
<       "incremental_size_ratio": "-0.00%",
<       "number_of_workers": 1,
28,29c23,27
<       "throughput_bytes": 5034634.655449181,
<       "timeline": 1
---
>       "throughput_bytes": 5034634,
>       "timeline": 1,
>       "wal_num": 2,
>       "wal_size": "32.0 MiB",
>       "wal_size_bytes": 33554432
31a30,31
>       "backup_type": "full",
>       "children_backup_ids": null,
37a40,43
>     "server_information": {
>       "data_checksums": null,
>       "summarize_wal": null
>     },
38a45
>     "system_id": "7382227563826631291",

I think this would also look fine?

barthisrael commented 1 month ago

I took a backup with rsync using Barman 3.10.1.

Backup 20240808T191838:
  Server Name            : support
  System Id              : 7382227563826631291
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data

  Base backup information:
    Disk usage           : 392.3 MiB (408.3 MiB with WALs)
    Incremental size     : 174.4 MiB (-55.54%)
    Timeline             : 1
    Begin WAL            : 000000010000000800000009
    End WAL              : 000000010000000800000009
    WAL number           : 1
    Begin time           : 2024-08-08 17:18:38.457024+00:00
    End time             : 2024-08-08 17:18:42.948175+00:00
    Copy time            : 3 seconds
    Estimated throughput : 53.2 MiB/s
    Begin Offset         : 40
    End Offset           : 344
    Begin LSN            : 8/9000028
    End LSN              : 8/9000158

  WAL information:
    No of files          : 0
    Disk usage           : 0 B
    Last available       : 000000010000000800000009

  Catalog information:
    Retention Policy     : VALID
    Previous Backup      : - (this is the oldest base backup)
    Next Backup          : - (this is the latest base backup)
Backup 20240808T191838:
  Server Name            : support
  System Id              : 7382227563826631291
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data

  Base backup information:
    Backup Method        : rsync-concurrent
    Backup Size          : 174.4 MiB (190.4 MiB with WALs)
    WAL Size             : 16.0 MiB
    Resource savings     : 217.9 MiB (55.54%)
    Timeline             : 1
    Begin WAL            : 000000010000000800000009
    End WAL              : 000000010000000800000009
    WAL number           : 1
    Begin time           : 2024-08-08 17:18:38.457024+00:00
    End time             : 2024-08-08 17:18:42.948175+00:00
    Copy time            : 3 seconds
    Estimated throughput : 53.2 MiB/s
    Begin Offset         : 40
    End Offset           : 344
    Begin LSN            : 8/9000028
    End LSN              : 8/9000158

  WAL information:
    No of files          : 0
    Disk usage           : 0 B
    Last available       : 000000010000000800000009

  Catalog information:
    Retention Policy     : VALID
    Previous Backup      : - (this is the oldest base backup)
    Next Backup          : - (this is the latest base backup)
9,10c9,12
<     Disk usage           : 392.3 MiB (408.3 MiB with WALs)
<     Incremental size     : 174.4 MiB (-55.54%)
---
>     Backup Method        : rsync-concurrent
>     Backup Size          : 174.4 MiB (190.4 MiB with WALs)
>     WAL Size             : 16.0 MiB
>     Resource savings     : 217.9 MiB (55.54%)

This looks fine.

{
  "support": {
    "backup_id": "20240808T191838",
    "base_backup_information": {
      "analysis_time": "less than one second",
      "analysis_time_seconds": 0.751278,
      "begin_lsn": "8/9000028",
      "begin_offset": 40,
      "begin_time": "2024-08-08 17:18:38.457024+00:00",
      "begin_time_timestamp": "1723137518",
      "begin_wal": "000000010000000800000009",
      "copy_time": "3 seconds",
      "copy_time_seconds": 3.275943,
      "disk_usage": "392.3 MiB",
      "disk_usage_bytes": 411323483,
      "disk_usage_with_wals": "408.3 MiB",
      "disk_usage_with_wals_bytes": 428100699,
      "end_lsn": "8/9000158",
      "end_offset": 344,
      "end_time": "2024-08-08 17:18:42.948175+00:00",
      "end_time_timestamp": "1723137522",
      "end_wal": "000000010000000800000009",
      "incremental_size": "174.4 MiB",
      "incremental_size_bytes": 182870921,
      "incremental_size_ratio": "-55.54%",
      "number_of_workers": 1,
      "throughput": "53.2 MiB/s",
      "throughput_bytes": 55822375.72509656,
      "timeline": 1
    },
    "catalog_information": {
      "next_backup": "- (this is the latest base backup)",
      "previous_backup": "- (this is the oldest base backup)",
      "retention_policy": "VALID"
    },
    "pgdata_directory": "/var/lib/pgsql/17/data",
    "postgresql_version": 170000,
    "status": "DONE",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "0 B",
      "disk_usage_bytes": 0,
      "last_available": "000000010000000800000009",
      "no_of_files": 0,
      "timelines": [],
      "wal_rate": 0,
      "wal_rate_per_second": 0
    }
  }
}

This breaks because of this:

{
  "_EXCEPTION": [
    "float() argument must be a string or a number, not 'NoneType'\nSee log file for more details."
  ],
  "support": {
    "backup_id": "20240808T191838",
    "server_information": {
      "data_checksums": null,
      "summarize_wal": null
    },
    "status": "DONE",
    "system_id": "7382227563826631291"
  }
}

If I work around that just to check how it would look like:

{
  "support": {
    "backup_id": "20240808T191838",
    "base_backup_information": {
      "backup_method": "rsync-concurrent",
      "backup_size": "174.4 MiB",
      "backup_size_bytes": 182870921,
      "backup_size_with_wals": "190.4 MiB",
      "backup_size_with_wals_bytes": 199648137,
      "begin_lsn": "8/9000028",
      "begin_offset": 40,
      "begin_time": "2024-08-08 17:18:38.457024+00:00",
      "begin_time_timestamp": "1723137518",
      "begin_wal": "000000010000000800000009",
      "copy_time": "3 seconds",
      "copy_time_seconds": 3.275943,
      "end_lsn": "8/9000158",
      "end_offset": 344,
      "end_time": "2024-08-08 17:18:42.948175+00:00",
      "end_time_timestamp": "1723137522",
      "end_wal": "000000010000000800000009",
      "resource_savings": "217.9 MiB",
      "resource_savings_bytes": 228452561,
      "resource_savings_percentage": "55.54%",
      "throughput": "53.2 MiB/s",
      "throughput_bytes": 55822375,
      "timeline": 1,
      "wal_num": 1,
      "wal_size": "16.0 MiB",
      "wal_size_bytes": 16777216
    },
    "catalog_information": {
      "next_backup": "- (this is the latest base backup)",
      "previous_backup": "- (this is the oldest base backup)",
      "retention_policy": "VALID"
    },
    "pgdata_directory": "/var/lib/pgsql/17/data",
    "postgresql_version": 170000,
    "server_information": {
      "data_checksums": null,
      "summarize_wal": null
    },
    "status": "DONE",
    "system_id": "7382227563826631291",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "0 B",
      "disk_usage_bytes": 0,
      "last_available": "000000010000000800000009",
      "no_of_files": 0,
      "timelines": [],
      "wal_rate": 0,
      "wal_rate_per_second": 0
    }
  }
}

And the diff would be:

5,6c5,9
<       "analysis_time": "less than one second",
<       "analysis_time_seconds": 0.751278,
---
>       "backup_method": "rsync-concurrent",
>       "backup_size": "174.4 MiB",
>       "backup_size_bytes": 182870921,
>       "backup_size_with_wals": "190.4 MiB",
>       "backup_size_with_wals_bytes": 199648137,
14,17d16
<       "disk_usage": "392.3 MiB",
<       "disk_usage_bytes": 411323483,
<       "disk_usage_with_wals": "408.3 MiB",
<       "disk_usage_with_wals_bytes": 428100699,
23,26c22,24
<       "incremental_size": "174.4 MiB",
<       "incremental_size_bytes": 182870921,
<       "incremental_size_ratio": "-55.54%",
<       "number_of_workers": 1,
---
>       "resource_savings": "217.9 MiB",
>       "resource_savings_bytes": 228452561,
>       "resource_savings_percentage": "55.54%",
28,29c26,30
<       "throughput_bytes": 55822375.72509656,
<       "timeline": 1
---
>       "throughput_bytes": 55822375,
>       "timeline": 1,
>       "wal_num": 1,
>       "wal_size": "16.0 MiB",
>       "wal_size_bytes": 16777216
37a39,42
>     "server_information": {
>       "data_checksums": null,
>       "summarize_wal": null
>     },
38a44
>     "system_id": "7382227563826631291",

I think this would also look fine?

barthisrael commented 1 month ago

I took a backup with rsync using this dev branch.

This is how console looks like:

Backup 20240808T195450:
  Server Name            : support
  System Id              : 7382227563826631291
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data
  Estimated Cluster Size : 392.2 MiB

  Server information:
    Checksums            : off
    WAL summarizer       : on

  Base backup information:
    Backup Method        : rsync-concurrent
    Backup Size          : 174.5 MiB (190.5 MiB with WALs)
    WAL Size             : 16.0 MiB
    Resource savings     : 217.8 MiB (55.53%)
    Timeline             : 1
    Begin WAL            : 000000010000000800000044
    End WAL              : 000000010000000800000044
    WAL number           : 1
    Begin time           : 2024-08-08 17:54:50.487781+00:00
    End time             : 2024-08-08 17:54:54.844055+00:00
    Copy time            : 2 seconds
    Estimated throughput : 62.7 MiB/s
    Begin Offset         : 1768
    End Offset           : 581232
    Begin LSN            : 8/440006E8
    End LSN              : 8/4408DE70

  WAL information:
    No of files          : 4
    Disk usage           : 64.0 MiB
    WAL rate             : 667.15/hour
    Last available       : 000000010000000800000048

  Catalog information:
    Retention Policy     : not enforced
    Previous Backup      : 20240808T195406
    Next Backup          : - (this is the latest base backup)

This is how JSON looks like:

{
  "support": {
    "backup_id": "20240808T195450",
    "base_backup_information": {
      "backup_method": "rsync-concurrent",
      "backup_size": "174.5 MiB",
      "backup_size_bytes": 182936568,
      "backup_size_with_wals": "190.5 MiB",
      "backup_size_with_wals_bytes": 199713784,
      "begin_lsn": "8/440006E8",
      "begin_offset": 1768,
      "begin_time": "2024-08-08 17:54:50.487781+00:00",
      "begin_time_timestamp": "1723139690",
      "begin_wal": "000000010000000800000044",
      "copy_time": "2 seconds",
      "copy_time_seconds": 2.782131,
      "end_lsn": "8/4408DE70",
      "end_offset": 581232,
      "end_time": "2024-08-08 17:54:54.844055+00:00",
      "end_time_timestamp": "1723139694",
      "end_wal": "000000010000000800000044",
      "resource_savings": "217.8 MiB",
      "resource_savings_bytes": 228351083,
      "resource_savings_percentage": "55.53%",
      "throughput": "62.7 MiB/s",
      "throughput_bytes": 65754117,
      "timeline": 1,
      "wal_num": 1,
      "wal_size": "16.0 MiB",
      "wal_size_bytes": 16777216
    },
    "catalog_information": {
      "next_backup": "- (this is the latest base backup)",
      "previous_backup": "20240808T195406",
      "retention_policy": "not enforced"
    },
    "cluster_size": "392.2 MiB",
    "cluster_size_bytes": 411206391,
    "pgdata_directory": "/var/lib/pgsql/17/data",
    "postgresql_version": 170000,
    "server_information": {
      "data_checksums": "off",
      "summarize_wal": "on"
    },
    "status": "DONE",
    "system_id": "7382227563826631291",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "64.0 MiB",
      "disk_usage_bytes": 67108864,
      "last_available": "000000010000000800000048",
      "no_of_files": 4,
      "timelines": [],
      "wal_rate": "667.15/hour",
      "wal_rate_per_second": 0.18531966221669852
    }
  }
}

That looks fine.

barthisrael commented 1 month ago

I took a pg_basebackup full backup with this dev branch.

This is the console output:

Backup 20240808T195400:
  Server Name            : pg17
  System Id              : 7382227563826631291
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data
  Estimated Cluster Size : 219.1 MiB

  Server information:
    Checksums            : off
    WAL summarizer       : on

  Base backup information:
    Backup Method        : postgres
    Backup Type          : full
    Backup Size          : 219.5 MiB (251.5 MiB with WALs)
    WAL Size             : 32.0 MiB
    Timeline             : 1
    Begin WAL            : 00000001000000080000002C
    End WAL              : 00000001000000080000002D
    WAL number           : 2
    Begin time           : 2024-08-08 17:54:00.636837+00:00
    End time             : 2024-08-08 17:54:03.149379+00:00
    Copy time            : 2 seconds
    Estimated throughput : 88.3 MiB/s
    Begin Offset         : 40
    End Offset           : 0
    Begin LSN            : 8/2C000028
    End LSN              : 8/2D000000

  WAL information:
    No of files          : 27
    Disk usage           : 432.0 MiB
    WAL rate             : 1296.86/hour
    Last available       : 000000010000000800000048

  Catalog information:
    Retention Policy     : VALID
    Previous Backup      : - (this is the oldest base backup)
    Next Backup          : 20240808T195509

    Children Backup(s)   : 20240808T195509

And the JSON output:

{
  "pg17": {
    "backup_id": "20240808T195400",
    "base_backup_information": {
      "backup_method": "postgres",
      "backup_size": "219.5 MiB",
      "backup_size_bytes": 230123508,
      "backup_size_with_wals": "251.5 MiB",
      "backup_size_with_wals_bytes": 263677940,
      "begin_lsn": "8/2C000028",
      "begin_offset": 40,
      "begin_time": "2024-08-08 17:54:00.636837+00:00",
      "begin_time_timestamp": "1723139640",
      "begin_wal": "00000001000000080000002C",
      "copy_time": "2 seconds",
      "copy_time_seconds": 2.485124,
      "end_lsn": "8/2D000000",
      "end_offset": 0,
      "end_time": "2024-08-08 17:54:03.149379+00:00",
      "end_time_timestamp": "1723139643",
      "end_wal": "00000001000000080000002D",
      "throughput": "88.3 MiB/s",
      "throughput_bytes": 92600412,
      "timeline": 1,
      "wal_num": 2,
      "wal_size": "32.0 MiB",
      "wal_size_bytes": 33554432
    },
    "catalog_information": {
      "backup_type": "full",
      "children_backup_ids": [
        "20240808T195509"
      ],
      "next_backup": "20240808T195509",
      "previous_backup": "- (this is the oldest base backup)",
      "retention_policy": "VALID"
    },
    "cluster_size": "219.1 MiB",
    "cluster_size_bytes": 229753591,
    "pgdata_directory": "/var/lib/pgsql/17/data",
    "postgresql_version": 170000,
    "server_information": {
      "data_checksums": "off",
      "summarize_wal": "on"
    },
    "status": "DONE",
    "system_id": "7382227563826631291",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "432.0 MiB",
      "disk_usage_bytes": 452984832,
      "last_available": "000000010000000800000048",
      "no_of_files": 27,
      "timelines": [],
      "wal_rate": "1296.86/hour",
      "wal_rate_per_second": 0.3602402104963426
    }
  }
}

They look fine, except for:

barthisrael commented 1 month ago

I took a pg_basebackup incremental backup with this dev branch.

This is the console output:

Backup 20240808T195509:
  Server Name            : pg17
  System Id              : 7382227563826631291
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data
  Estimated Cluster Size : 392.2 MiB

  Server information:
    Checksums            : off
    WAL summarizer       : on

  Base backup information:
    Backup Method        : postgres
    Backup Type          : incremental
    Backup Size          : 176.7 MiB (208.7 MiB with WALs)
    WAL Size             : 32.0 MiB
    Timeline             : 1
    Begin WAL            : 000000010000000800000047
    End WAL              : 000000010000000800000048
    WAL number           : 2
    Begin time           : 2024-08-08 17:55:09.440502+00:00
    End time             : 2024-08-08 17:55:12.028233+00:00
    Copy time            : 2 seconds
    Estimated throughput : 68.8 MiB/s
    Begin Offset         : 40
    End Offset           : 0
    Begin LSN            : 8/47000028
    End LSN              : 8/48000000

  WAL information:
    No of files          : 0
    Disk usage           : 0 B
    WAL rate             : 624.43/hour
    Last available       : 000000010000000800000048

  Catalog information:
    Retention Policy     : VALID
    Previous Backup      : 20240808T195400
    Next Backup          : - (this is the latest base backup)

    Root Backup          : 20240808T195400
    Parent Backup        : 20240808T195400
    Backup chain size    : 2

And the JSON output:

{
  "pg17": {
    "backup_id": "20240808T195509",
    "base_backup_information": {
      "backup_method": "postgres",
      "backup_size": "176.7 MiB",
      "backup_size_bytes": 185282555,
      "backup_size_with_wals": "208.7 MiB",
      "backup_size_with_wals_bytes": 218836987,
      "begin_lsn": "8/47000028",
      "begin_offset": 40,
      "begin_time": "2024-08-08 17:55:09.440502+00:00",
      "begin_time_timestamp": "1723139709",
      "begin_wal": "000000010000000800000047",
      "copy_time": "2 seconds",
      "copy_time_seconds": 2.567636,
      "end_lsn": "8/48000000",
      "end_offset": 0,
      "end_time": "2024-08-08 17:55:12.028233+00:00",
      "end_time_timestamp": "1723139712",
      "end_wal": "000000010000000800000048",
      "resource_savings": "0 B",
      "resource_savings_bytes": 0,
      "resource_savings_percentage": "0.00%",
      "throughput": "68.8 MiB/s",
      "throughput_bytes": 72160756,
      "timeline": 1,
      "wal_num": 2,
      "wal_size": "32.0 MiB",
      "wal_size_bytes": 33554432
    },
    "catalog_information": {
      "backup_type": "incremental",
      "chain_size": 2,
      "children_backup_ids": null,
      "next_backup": "- (this is the latest base backup)",
      "parent_backup_id": "20240808T195400",
      "previous_backup": "20240808T195400",
      "retention_policy": "VALID",
      "root_backup_id": "20240808T195400"
    },
    "cluster_size": "392.2 MiB",
    "cluster_size_bytes": 411239159,
    "pgdata_directory": "/var/lib/pgsql/17/data",
    "postgresql_version": 170000,
    "server_information": {
      "data_checksums": "off",
      "summarize_wal": "on"
    },
    "status": "DONE",
    "system_id": "7382227563826631291",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "0 B",
      "disk_usage_bytes": 0,
      "last_available": "000000010000000800000048",
      "no_of_files": 0,
      "timelines": [],
      "wal_rate": "624.43/hour",
      "wal_rate_per_second": 0.17345351541820306
    }
  }
}

I have similar comments to the ones I posted for the full backup in the previous PR comment. Besides that, I'm missing "resource savings" in the console output.

barthisrael commented 1 month ago

To sum up:

barthisrael commented 1 month ago

barman-cloud-backup-show fails BTW:

$ barman-cloud-backup-show --endpoint-url=http://127.0.0.1:4566 -P barman-cloud s3://barman-s3-test pg17-cloud 20240812T212922
Backup 20240812T212922:
  Server Name            : cloud
  System Id              : 7400339383961701111
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /home/vagrant/pgdata-for-cloud
  Estimated Cluster Size : 22.4 MiB

  Server information:
    Checksums            : off
    WAL summarizer       : off

  Base backup information:
Traceback (most recent call last):
  File "/home/vagrant/Repositories/edb/barman/barman/clients/cloud_backup_show.py", line 78, in main
    ConsoleOutputWriter.render_show_backup(backup_info.to_dict(), print)
  File "/home/vagrant/Repositories/edb/barman/barman/output.py", line 1064, in render_show_backup
    ConsoleOutputWriter.render_show_backup_base(
  File "/home/vagrant/Repositories/edb/barman/barman/output.py", line 839, in render_show_backup_base
    backup_size_output = "{}".format(pretty_size(backup_size))
  File "/home/vagrant/Repositories/edb/barman/barman/utils.py", line 178, in pretty_size
    size = float(size)
TypeError: float() argument must be a string or a number, not 'NoneType'
2024-08-12 21:30:49,171 [1051013] ERROR: Barman cloud backup show exception: float() argument must be a string or a number, not 'NoneType'

FWIW the original one didn't have a size:

$ barman-cloud-backup-show --endpoint-url=http://127.0.0.1:4566 -P barman-cloud s3://barman-s3-test pg17-cloud 20240812T212403
Backup 20240812T212403:
  Server Name            : cloud
  System Id              : 7400339383961701111
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /home/vagrant/pgdata-for-cloud

  Base backup information:
    Timeline             : 1
    Begin WAL            : 000000010000000000000004
    End WAL              : 000000010000000000000004
    Begin time           : 2024-08-12 21:24:03.992578+02:00
    End time             : 2024-08-12 21:24:05.539279+02:00
    Copy time            : less than one second
    Begin Offset         : 96
    End Offset           : 344
    Begin LSN            : 0/4000060
    End LSN              : 0/4000158
barthisrael commented 1 month ago

Backups taken with 3.10.1

Postgres method

Show as console with 3.10.1

Backup 20240813T204536:
  Backup Name            : 3.10.1_postgres
  Server Name            : postgres
  System Id              : 7402632902805533789
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data

  Base backup information:
    Disk usage           : 22.3 MiB (54.3 MiB with WALs)
    Incremental size     : 22.3 MiB (-0.00%)
    Timeline             : 1
    Begin WAL            : 000000010000000000000027
    End WAL              : 000000010000000000000028
    WAL number           : 2
    Begin time           : 2024-08-13 20:45:36.130288+02:00
    End time             : 2024-08-13 20:45:37.635507+02:00
    Copy time            : 1 second
    Estimated throughput : 15.0 MiB/s
    Begin Offset         : 40
    End Offset           : 0
    Begin LSN            : 0/27000028
    End LSN              : 0/28000000

  WAL information:
    No of files          : 1
    Disk usage           : 16.0 MiB
    WAL rate             : 794.76/hour
    Last available       : 000000010000000000000029

  Catalog information:
    Retention Policy     : not enforced
    Previous Backup      : - (this is the oldest base backup)
    Next Backup          : - (this is the latest base backup)

Show as console with dev

Backup 20240813T204536:
  Backup Name            : 3.10.1_postgres
  Server Name            : postgres
  System Id              : 7402632902805533789
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data

  Base backup information:
    Backup Method        : postgres
    Backup Type          : full
    Backup Size          : 22.3 MiB (54.3 MiB with WALs)
    WAL Size             : 32.0 MiB
    Timeline             : 1
    Begin WAL            : 000000010000000000000027
    End WAL              : 000000010000000000000028
    WAL number           : 2
    Begin time           : 2024-08-13 20:45:36.130288+02:00
    End time             : 2024-08-13 20:45:37.635507+02:00
    Copy time            : 1 second
    Estimated throughput : 15.0 MiB/s
    Begin Offset         : 40
    End Offset           : 0
    Begin LSN            : 0/27000028
    End LSN              : 0/28000000

  WAL information:
    No of files          : 1
    Disk usage           : 16.0 MiB
    WAL rate             : 794.76/hour
    Last available       : 000000010000000000000029

  Catalog information:
    Retention Policy     : not enforced
    Previous Backup      : - (this is the oldest base backup)
    Next Backup          : - (this is the latest base backup)

Diff as console

10,11c10,13
<     Disk usage           : 22.3 MiB (54.3 MiB with WALs)
<     Incremental size     : 22.3 MiB (-0.00%)
---
>     Backup Method        : postgres
>     Backup Type          : full
>     Backup Size          : 22.3 MiB (54.3 MiB with WALs)
>     WAL Size             : 32.0 MiB

Show as JSON with 3.10.1

{
  "postgres": {
    "backup_id": "20240813T204536",
    "backup_name": "3.10.1_postgres",
    "base_backup_information": {
      "analysis_time": "less than one second",
      "analysis_time_seconds": 0,
      "begin_lsn": "0/27000028",
      "begin_offset": 40,
      "begin_time": "2024-08-13 20:45:36.130288+02:00",
      "begin_time_timestamp": "1723574736",
      "begin_wal": "000000010000000000000027",
      "copy_time": "1 second",
      "copy_time_seconds": 1.485377,
      "disk_usage": "22.3 MiB",
      "disk_usage_bytes": 23389052,
      "disk_usage_with_wals": "54.3 MiB",
      "disk_usage_with_wals_bytes": 56943484,
      "end_lsn": "0/28000000",
      "end_offset": 0,
      "end_time": "2024-08-13 20:45:37.635507+02:00",
      "end_time_timestamp": "1723574737",
      "end_wal": "000000010000000000000028",
      "incremental_size": "22.3 MiB",
      "incremental_size_bytes": 23389052,
      "incremental_size_ratio": "-0.00%",
      "number_of_workers": 1,
      "throughput": "15.0 MiB/s",
      "throughput_bytes": 15746205.845384708,
      "timeline": 1
    },
    "catalog_information": {
      "next_backup": "- (this is the latest base backup)",
      "previous_backup": "- (this is the oldest base backup)",
      "retention_policy": "not enforced"
    },
    "pgdata_directory": "/var/lib/pgsql/17/data",
    "postgresql_version": 170000,
    "status": "DONE",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "16.0 MiB",
      "disk_usage_bytes": 16777216,
      "last_available": "000000010000000000000029",
      "no_of_files": 1,
      "timelines": [],
      "wal_rate": "794.76/hour",
      "wal_rate_per_second": 0.22076662259023117
    }
  }
}

Show as JSON with dev

{
  "postgres": {
    "backup_id": "20240813T204536",
    "backup_name": "3.10.1_postgres",
    "backup_type": "full",
    "base_backup_information": {
      "backup_method": "postgres",
      "backup_size": "22.3 MiB",
      "backup_size_bytes": 23389052,
      "backup_size_with_wals": "54.3 MiB",
      "backup_size_with_wals_bytes": 56943484,
      "begin_lsn": "0/27000028",
      "begin_offset": 40,
      "begin_time": "2024-08-13 20:45:36.130288+02:00",
      "begin_time_timestamp": "1723574736",
      "begin_wal": "000000010000000000000027",
      "copy_time": "1 second",
      "copy_time_seconds": 1.485377,
      "end_lsn": "0/28000000",
      "end_offset": 0,
      "end_time": "2024-08-13 20:45:37.635507+02:00",
      "end_time_timestamp": "1723574737",
      "end_wal": "000000010000000000000028",
      "throughput": "15.0 MiB/s",
      "throughput_bytes": 15746205,
      "timeline": 1,
      "wal_num": 2,
      "wal_size": "32.0 MiB",
      "wal_size_bytes": 33554432
    },
    "catalog_information": {
      "children_backup_ids": null,
      "next_backup": "- (this is the latest base backup)",
      "previous_backup": "- (this is the oldest base backup)",
      "retention_policy": "not enforced"
    },
    "pgdata_directory": "/var/lib/pgsql/17/data",
    "postgresql_version": 170000,
    "server_information": {
      "data_checksums": null,
      "summarize_wal": null
    },
    "status": "DONE",
    "system_id": "7402632902805533789",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "16.0 MiB",
      "disk_usage_bytes": 16777216,
      "last_available": "000000010000000000000029",
      "no_of_files": 1,
      "timelines": [],
      "wal_rate": "794.76/hour",
      "wal_rate_per_second": 0.22076662259023117
    }
  }
}

Diff as JSON

4a5
>     "backup_type": "full",
6,7c7,11
<       "analysis_time": "less than one second",
<       "analysis_time_seconds": 0,
---
>       "backup_method": "postgres",
>       "backup_size": "22.3 MiB",
>       "backup_size_bytes": 23389052,
>       "backup_size_with_wals": "54.3 MiB",
>       "backup_size_with_wals_bytes": 56943484,
15,18d18
<       "disk_usage": "22.3 MiB",
<       "disk_usage_bytes": 23389052,
<       "disk_usage_with_wals": "54.3 MiB",
<       "disk_usage_with_wals_bytes": 56943484,
24,27d23
<       "incremental_size": "22.3 MiB",
<       "incremental_size_bytes": 23389052,
<       "incremental_size_ratio": "-0.00%",
<       "number_of_workers": 1,
29,30c25,29
<       "throughput_bytes": 15746205.845384708,
<       "timeline": 1
---
>       "throughput_bytes": 15746205,
>       "timeline": 1,
>       "wal_num": 2,
>       "wal_size": "32.0 MiB",
>       "wal_size_bytes": 33554432
32a32
>       "children_backup_ids": null,
38a39,42
>     "server_information": {
>       "data_checksums": null,
>       "summarize_wal": null
>     },
39a44
>     "system_id": "7402632902805533789",

Rsync method

Show as console with 3.10.1

Backup 20240813T204547:
  Backup Name            : 3.10.1_rsync
  Server Name            : rsync
  System Id              : 7402632902805533789
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data

  Base backup information:
    Disk usage           : 22.1 MiB (38.1 MiB with WALs)
    Incremental size     : 22.1 MiB (-0.00%)
    Timeline             : 1
    Begin WAL            : 000000010000000000000029
    End WAL              : 000000010000000000000029
    WAL number           : 1
    Begin time           : 2024-08-13 20:45:47.711858+02:00
    End time             : 2024-08-13 20:45:50.302078+02:00
    Copy time            : 1 second
    Estimated throughput : 12.8 MiB/s
    Begin Offset         : 40
    End Offset           : 288
    Begin LSN            : 0/29000028
    End LSN              : 0/29000120

  WAL information:
    No of files          : 0
    Disk usage           : 0 B
    Last available       : 000000010000000000000029

  Catalog information:
    Retention Policy     : not enforced
    Previous Backup      : - (this is the oldest base backup)
    Next Backup          : - (this is the latest base backup)

Show as console with dev

Backup 20240813T204547:
  Backup Name            : 3.10.1_rsync
  Server Name            : rsync
  System Id              : 7402632902805533789
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data

  Base backup information:
    Backup Method        : rsync-concurrent
    Backup Size          : 22.1 MiB (38.1 MiB with WALs)
    WAL Size             : 16.0 MiB
    Resource savings     : 0 B (0.00%)
    Timeline             : 1
    Begin WAL            : 000000010000000000000029
    End WAL              : 000000010000000000000029
    WAL number           : 1
    Begin time           : 2024-08-13 20:45:47.711858+02:00
    End time             : 2024-08-13 20:45:50.302078+02:00
    Copy time            : 1 second
    Estimated throughput : 12.8 MiB/s
    Begin Offset         : 40
    End Offset           : 288
    Begin LSN            : 0/29000028
    End LSN              : 0/29000120

  WAL information:
    No of files          : 0
    Disk usage           : 0 B
    Last available       : 000000010000000000000029

  Catalog information:
    Retention Policy     : not enforced
    Previous Backup      : - (this is the oldest base backup)
    Next Backup          : - (this is the latest base backup)

Diff as console

10,11c10,13
<     Disk usage           : 22.1 MiB (38.1 MiB with WALs)
<     Incremental size     : 22.1 MiB (-0.00%)
---
>     Backup Method        : rsync-concurrent
>     Backup Size          : 22.1 MiB (38.1 MiB with WALs)
>     WAL Size             : 16.0 MiB
>     Resource savings     : 0 B (0.00%)

Show as JSON with 3.10.1

{
  "rsync": {
    "backup_id": "20240813T204547",
    "backup_name": "3.10.1_rsync",
    "base_backup_information": {
      "analysis_time": "less than one second",
      "analysis_time_seconds": 0.630416,
      "begin_lsn": "0/29000028",
      "begin_offset": 40,
      "begin_time": "2024-08-13 20:45:47.711858+02:00",
      "begin_time_timestamp": "1723574747",
      "begin_wal": "000000010000000000000029",
      "copy_time": "1 second",
      "copy_time_seconds": 1.724801,
      "disk_usage": "22.1 MiB",
      "disk_usage_bytes": 23225366,
      "disk_usage_with_wals": "38.1 MiB",
      "disk_usage_with_wals_bytes": 40002582,
      "end_lsn": "0/29000120",
      "end_offset": 288,
      "end_time": "2024-08-13 20:45:50.302078+02:00",
      "end_time_timestamp": "1723574750",
      "end_wal": "000000010000000000000029",
      "incremental_size": "22.1 MiB",
      "incremental_size_bytes": 23225366,
      "incremental_size_ratio": "-0.00%",
      "number_of_workers": 1,
      "throughput": "12.8 MiB/s",
      "throughput_bytes": 13465533.705047712,
      "timeline": 1
    },
    "catalog_information": {
      "next_backup": "- (this is the latest base backup)",
      "previous_backup": "- (this is the oldest base backup)",
      "retention_policy": "not enforced"
    },
    "pgdata_directory": "/var/lib/pgsql/17/data",
    "postgresql_version": 170000,
    "status": "DONE",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "0 B",
      "disk_usage_bytes": 0,
      "last_available": "000000010000000000000029",
      "no_of_files": 0,
      "timelines": [],
      "wal_rate": 0,
      "wal_rate_per_second": 0
    }
  }
}

Show as JSON with dev

{
  "rsync": {
    "backup_id": "20240813T204547",
    "backup_name": "3.10.1_rsync",
    "backup_type": "rsync",
    "base_backup_information": {
      "backup_method": "rsync-concurrent",
      "backup_size": "22.1 MiB",
      "backup_size_bytes": 23225366,
      "backup_size_with_wals": "38.1 MiB",
      "backup_size_with_wals_bytes": 40002582,
      "begin_lsn": "0/29000028",
      "begin_offset": 40,
      "begin_time": "2024-08-13 20:45:47.711858+02:00",
      "begin_time_timestamp": "1723574747",
      "begin_wal": "000000010000000000000029",
      "copy_time": "1 second",
      "copy_time_seconds": 1.724801,
      "end_lsn": "0/29000120",
      "end_offset": 288,
      "end_time": "2024-08-13 20:45:50.302078+02:00",
      "end_time_timestamp": "1723574750",
      "end_wal": "000000010000000000000029",
      "resource_savings": "0 B",
      "resource_savings_bytes": 0,
      "resource_savings_percentage": "0.00%",
      "throughput": "12.8 MiB/s",
      "throughput_bytes": 13465533,
      "timeline": 1,
      "wal_num": 1,
      "wal_size": "16.0 MiB",
      "wal_size_bytes": 16777216
    },
    "catalog_information": {
      "next_backup": "- (this is the latest base backup)",
      "previous_backup": "- (this is the oldest base backup)",
      "retention_policy": "not enforced"
    },
    "pgdata_directory": "/var/lib/pgsql/17/data",
    "postgresql_version": 170000,
    "server_information": {
      "data_checksums": null,
      "summarize_wal": null
    },
    "status": "DONE",
    "system_id": "7402632902805533789",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "0 B",
      "disk_usage_bytes": 0,
      "last_available": "000000010000000000000029",
      "no_of_files": 0,
      "timelines": [],
      "wal_rate": 0,
      "wal_rate_per_second": 0
    }
  }
}

Diff as JSON

4a5
>     "backup_type": "rsync",
6,7c7,11
<       "analysis_time": "less than one second",
<       "analysis_time_seconds": 0.630416,
---
>       "backup_method": "rsync-concurrent",
>       "backup_size": "22.1 MiB",
>       "backup_size_bytes": 23225366,
>       "backup_size_with_wals": "38.1 MiB",
>       "backup_size_with_wals_bytes": 40002582,
15,18d18
<       "disk_usage": "22.1 MiB",
<       "disk_usage_bytes": 23225366,
<       "disk_usage_with_wals": "38.1 MiB",
<       "disk_usage_with_wals_bytes": 40002582,
24,27c24,26
<       "incremental_size": "22.1 MiB",
<       "incremental_size_bytes": 23225366,
<       "incremental_size_ratio": "-0.00%",
<       "number_of_workers": 1,
---
>       "resource_savings": "0 B",
>       "resource_savings_bytes": 0,
>       "resource_savings_percentage": "0.00%",
29,30c28,32
<       "throughput_bytes": 13465533.705047712,
<       "timeline": 1
---
>       "throughput_bytes": 13465533,
>       "timeline": 1,
>       "wal_num": 1,
>       "wal_size": "16.0 MiB",
>       "wal_size_bytes": 16777216
38a41,44
>     "server_information": {
>       "data_checksums": null,
>       "summarize_wal": null
>     },
39a46
>     "system_id": "7402632902805533789",

Snapshot method

Show as console with 3.10.1

Backup 20240813T204643:
  Backup Name            : 3.10.1_snapshot
  Server Name            : snapshot
  System Id              : 7402620047885836080
  Status                 : DONE
  PostgreSQL Version     : 160004
  PGDATA directory       : /opt/postgres/data

  Snapshot information:
    provider             : aws
    account_id           : 714574844897
    region               : sa-east-1

    device_name          : /dev/sdf
    snapshot_id          : snap-054c513ccbd9608a1
    snapshot_name        : Barman_AWS:1:/dev/sdf-20240813t204643
    Mount point          : /opt/postgres
    Mount options        : rw,noatime,seclabel

  Base backup information:
    Disk usage           : 970 B (16.0 MiB with WALs)
    Incremental size     : 970 B (-0.00%)
    Timeline             : 1
    Begin WAL            : 000000010000000000000012
    End WAL              : 000000010000000000000012
    WAL number           : 1
    Begin time           : 2024-08-13 18:46:43.433728+00:00
    End time             : 2024-08-13 18:47:45.864193+00:00
    Copy time            : 1 minute, 2 seconds
    Estimated throughput : 15 B/s
    Begin Offset         : 40
    End Offset           : 312
    Begin LSN            : 0/12000028
    End LSN              : 0/12000138

  WAL information:
    No of files          : 0
    Disk usage           : 0 B
    Last available       : 000000010000000000000012

  Catalog information:
    Retention Policy     : not enforced
    Previous Backup      : - (this is the oldest base backup)
    Next Backup          : - (this is the latest base backup)

Show as console with dev

Backup 20240813T204643:
  Backup Name            : 3.10.1_snapshot
  Server Name            : snapshot
  System Id              : 7402620047885836080
  Status                 : DONE
  PostgreSQL Version     : 160004
  PGDATA directory       : /opt/postgres/data

  Snapshot information:
    provider             : aws
    account_id           : 714574844897
    region               : sa-east-1

    device_name          : /dev/sdf
    snapshot_id          : snap-054c513ccbd9608a1
    snapshot_name        : Barman_AWS:1:/dev/sdf-20240813t204643
    Mount point          : /opt/postgres
    Mount options        : rw,noatime,seclabel

  Base backup information:
    Backup Method        : snapshot-concurrent
    Backup Size          : 970 B (16.0 MiB with WALs)
    WAL Size             : 16.0 MiB
    Resource savings     : 0 B (0.00%)
    Timeline             : 1
    Begin WAL            : 000000010000000000000012
    End WAL              : 000000010000000000000012
    WAL number           : 1
    Begin time           : 2024-08-13 18:46:43.433728+00:00
    End time             : 2024-08-13 18:47:45.864193+00:00
    Copy time            : 1 minute, 2 seconds
    Estimated throughput : 15 B/s
    Begin Offset         : 40
    End Offset           : 312
    Begin LSN            : 0/12000028
    End LSN              : 0/12000138

  WAL information:
    No of files          : 0
    Disk usage           : 0 B
    Last available       : 000000010000000000000012

  Catalog information:
    Retention Policy     : not enforced
    Previous Backup      : - (this is the oldest base backup)
    Next Backup          : - (this is the latest base backup)

Diff as console

21,22c21,24
<     Disk usage           : 970 B (16.0 MiB with WALs)
<     Incremental size     : 970 B (-0.00%)
---
>     Backup Method        : snapshot-concurrent
>     Backup Size          : 970 B (16.0 MiB with WALs)
>     WAL Size             : 16.0 MiB
>     Resource savings     : 0 B (0.00%)

Show as JSON with 3.10.1

{
  "snapshot": {
    "backup_id": "20240813T204643",
    "backup_name": "3.10.1_snapshot",
    "base_backup_information": {
      "analysis_time": "less than one second",
      "analysis_time_seconds": 0,
      "begin_lsn": "0/12000028",
      "begin_offset": 40,
      "begin_time": "2024-08-13 18:46:43.433728+00:00",
      "begin_time_timestamp": "1723574803",
      "begin_wal": "000000010000000000000012",
      "copy_time": "1 minute, 2 seconds",
      "copy_time_seconds": 62.018889,
      "disk_usage": "970 B",
      "disk_usage_bytes": 970,
      "disk_usage_with_wals": "16.0 MiB",
      "disk_usage_with_wals_bytes": 16778186,
      "end_lsn": "0/12000138",
      "end_offset": 312,
      "end_time": "2024-08-13 18:47:45.864193+00:00",
      "end_time_timestamp": "1723574865",
      "end_wal": "000000010000000000000012",
      "incremental_size": "970 B",
      "incremental_size_bytes": 970,
      "incremental_size_ratio": "-0.00%",
      "number_of_workers": 1,
      "throughput": "15 B/s",
      "throughput_bytes": 15.640396267014715,
      "timeline": 1
    },
    "catalog_information": {
      "next_backup": "- (this is the latest base backup)",
      "previous_backup": "- (this is the oldest base backup)",
      "retention_policy": "not enforced"
    },
    "pgdata_directory": "/opt/postgres/data",
    "postgresql_version": 160004,
    "snapshots_info": {
      "provider": "aws",
      "provider_info": {
        "account_id": "714574844897",
        "region": "sa-east-1"
      },
      "snapshots": [
        {
          "mount": {
            "mount_options": "rw,noatime,seclabel",
            "mount_point": "/opt/postgres"
          },
          "provider": {
            "device_name": "/dev/sdf",
            "snapshot_id": "snap-054c513ccbd9608a1",
            "snapshot_name": "Barman_AWS:1:/dev/sdf-20240813t204643"
          }
        }
      ]
    },
    "status": "DONE",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "0 B",
      "disk_usage_bytes": 0,
      "last_available": "000000010000000000000012",
      "no_of_files": 0,
      "timelines": [],
      "wal_rate": 0,
      "wal_rate_per_second": 0
    }
  }
}

Show as JSON with dev

{
  "snapshot": {
    "backup_id": "20240813T204643",
    "backup_name": "3.10.1_snapshot",
    "backup_type": "rsync",
    "base_backup_information": {
      "backup_method": "snapshot-concurrent",
      "backup_size": "970 B",
      "backup_size_bytes": 970,
      "backup_size_with_wals": "16.0 MiB",
      "backup_size_with_wals_bytes": 16778186,
      "begin_lsn": "0/12000028",
      "begin_offset": 40,
      "begin_time": "2024-08-13 18:46:43.433728+00:00",
      "begin_time_timestamp": "1723574803",
      "begin_wal": "000000010000000000000012",
      "copy_time": "1 minute, 2 seconds",
      "copy_time_seconds": 62.018889,
      "end_lsn": "0/12000138",
      "end_offset": 312,
      "end_time": "2024-08-13 18:47:45.864193+00:00",
      "end_time_timestamp": "1723574865",
      "end_wal": "000000010000000000000012",
      "resource_savings": "0 B",
      "resource_savings_bytes": 0,
      "resource_savings_percentage": "0.00%",
      "throughput": "15 B/s",
      "throughput_bytes": 15,
      "timeline": 1,
      "wal_num": 1,
      "wal_size": "16.0 MiB",
      "wal_size_bytes": 16777216
    },
    "catalog_information": {
      "next_backup": "- (this is the latest base backup)",
      "previous_backup": "- (this is the oldest base backup)",
      "retention_policy": "not enforced"
    },
    "pgdata_directory": "/opt/postgres/data",
    "postgresql_version": 160004,
    "server_information": {
      "data_checksums": null,
      "summarize_wal": null
    },
    "snapshots_info": {
      "provider": "aws",
      "provider_info": {
        "account_id": "714574844897",
        "region": "sa-east-1"
      },
      "snapshots": [
        {
          "mount": {
            "mount_options": "rw,noatime,seclabel",
            "mount_point": "/opt/postgres"
          },
          "provider": {
            "device_name": "/dev/sdf",
            "snapshot_id": "snap-054c513ccbd9608a1",
            "snapshot_name": "Barman_AWS:1:/dev/sdf-20240813t204643"
          }
        }
      ]
    },
    "status": "DONE",
    "system_id": "7402620047885836080",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "0 B",
      "disk_usage_bytes": 0,
      "last_available": "000000010000000000000012",
      "no_of_files": 0,
      "timelines": [],
      "wal_rate": 0,
      "wal_rate_per_second": 0
    }
  }
}

Diff as JSON

4a5
>     "backup_type": "rsync",
6,7c7,11
<       "analysis_time": "less than one second",
<       "analysis_time_seconds": 0,
---
>       "backup_method": "snapshot-concurrent",
>       "backup_size": "970 B",
>       "backup_size_bytes": 970,
>       "backup_size_with_wals": "16.0 MiB",
>       "backup_size_with_wals_bytes": 16778186,
15,18d18
<       "disk_usage": "970 B",
<       "disk_usage_bytes": 970,
<       "disk_usage_with_wals": "16.0 MiB",
<       "disk_usage_with_wals_bytes": 16778186,
24,27c24,26
<       "incremental_size": "970 B",
<       "incremental_size_bytes": 970,
<       "incremental_size_ratio": "-0.00%",
<       "number_of_workers": 1,
---
>       "resource_savings": "0 B",
>       "resource_savings_bytes": 0,
>       "resource_savings_percentage": "0.00%",
29,30c28,32
<       "throughput_bytes": 15.640396267014715,
<       "timeline": 1
---
>       "throughput_bytes": 15,
>       "timeline": 1,
>       "wal_num": 1,
>       "wal_size": "16.0 MiB",
>       "wal_size_bytes": 16777216
38a41,44
>     "server_information": {
>       "data_checksums": null,
>       "summarize_wal": null
>     },
59a66
>     "system_id": "7402620047885836080",

Cloud backup

Show as console with 3.10.1

Backup 20240813T205658:
  Backup Name            : 3.10.1_cloud
  Server Name            : cloud
  System Id              : 7402692746266015596
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /home/vagrant/pgdata-for-cloud

  Base backup information:
    Timeline             : 1
    Begin WAL            : 000000010000000000000004
    End WAL              : 000000010000000000000004
    Begin time           : 2024-08-13 20:56:58.212780+02:00
    End time             : 2024-08-13 20:56:59.028511+02:00
    Copy time            : less than one second
    Begin Offset         : 40
    End Offset           : 288
    Begin LSN            : 0/4000028
    End LSN              : 0/4000120

Show as console with dev

Backup 20240813T205658:
  Backup Name            : 3.10.1_cloud
  Server Name            : cloud
  System Id              : 7402692746266015596
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /home/vagrant/pgdata-for-cloud

  Base backup information:
    Timeline             : 1
    Begin WAL            : 000000010000000000000004
    End WAL              : 000000010000000000000004
    Begin time           : 2024-08-13 20:56:58.212780+02:00
    End time             : 2024-08-13 20:56:59.028511+02:00
    Copy time            : less than one second
    Begin Offset         : 40
    End Offset           : 288
    Begin LSN            : 0/4000028
    End LSN              : 0/4000120

Diff as console

Show as JSON with 3.10.1

{
  "cloud": {
    "backup_label": "'START WAL LOCATION: 0/4000028 (file 000000010000000000000004)\\nCHECKPOINT LOCATION: 0/4000080\\nBACKUP METHOD: streamed\\nBACKUP FROM: primary\\nSTART TIME: 2024-08-13 20:56:58 CEST\\nLABEL: Barman backup cloud 20240813T205658\\nSTART TIMELINE: 1\\n'",
    "backup_name": "3.10.1_cloud",
    "begin_offset": 40,
    "begin_time": "Tue Aug 13 20:56:58 2024",
    "begin_wal": "000000010000000000000004",
    "begin_xlog": "0/4000028",
    "compression": null,
    "config_file": "/home/vagrant/pgdata-for-cloud/postgresql.conf",
    "copy_stats": {
      "total_time": 1.203094,
      "number_of_workers": 2,
      "analysis_time": 0,
      "analysis_time_per_item": {
        "data": 0
      },
      "copy_time_per_item": {
        "data": 0.54363
      },
      "serialized_copy_time_per_item": {
        "data": 0.638019
      },
      "copy_time": 0.54363,
      "serialized_copy_time": 0.638019
    },
    "deduplicated_size": null,
    "end_offset": 288,
    "end_time": "Tue Aug 13 20:56:59 2024",
    "end_wal": "000000010000000000000004",
    "end_xlog": "0/4000120",
    "error": null,
    "hba_file": "/home/vagrant/pgdata-for-cloud/pg_hba.conf",
    "ident_file": "/home/vagrant/pgdata-for-cloud/pg_ident.conf",
    "included_files": null,
    "mode": null,
    "pgdata": "/home/vagrant/pgdata-for-cloud",
    "server_name": "cloud",
    "size": null,
    "status": "DONE",
    "systemid": "7402692746266015596",
    "tablespaces": null,
    "timeline": 1,
    "version": 170000,
    "xlog_segment_size": 16777216,
    "backup_id": "20240813T205658"
  }
}

Show as JSON with dev

{
  "cloud": {
    "backup_label": "'START WAL LOCATION: 0/4000028 (file 000000010000000000000004)\\nCHECKPOINT LOCATION: 0/4000080\\nBACKUP METHOD: streamed\\nBACKUP FROM: primary\\nSTART TIME: 2024-08-13 20:56:58 CEST\\nLABEL: Barman backup cloud 20240813T205658\\nSTART TIMELINE: 1\\n'",
    "backup_name": "3.10.1_cloud",
    "begin_offset": 40,
    "begin_time": "Tue Aug 13 20:56:58 2024",
    "begin_wal": "000000010000000000000004",
    "begin_xlog": "0/4000028",
    "children_backup_ids": null,
    "cluster_size": null,
    "compression": null,
    "config_file": "/home/vagrant/pgdata-for-cloud/postgresql.conf",
    "copy_stats": {
      "total_time": 1.203094,
      "number_of_workers": 2,
      "analysis_time": 0,
      "analysis_time_per_item": {
        "data": 0
      },
      "copy_time_per_item": {
        "data": 0.54363
      },
      "serialized_copy_time_per_item": {
        "data": 0.638019
      },
      "copy_time": 0.54363,
      "serialized_copy_time": 0.638019
    },
    "data_checksums": null,
    "deduplicated_size": null,
    "end_offset": 288,
    "end_time": "Tue Aug 13 20:56:59 2024",
    "end_wal": "000000010000000000000004",
    "end_xlog": "0/4000120",
    "error": null,
    "hba_file": "/home/vagrant/pgdata-for-cloud/pg_hba.conf",
    "ident_file": "/home/vagrant/pgdata-for-cloud/pg_ident.conf",
    "included_files": null,
    "mode": null,
    "parent_backup_id": null,
    "pgdata": "/home/vagrant/pgdata-for-cloud",
    "server_name": "cloud",
    "size": null,
    "status": "DONE",
    "summarize_wal": null,
    "systemid": "7402692746266015596",
    "tablespaces": null,
    "timeline": 1,
    "version": 170000,
    "xlog_segment_size": 16777216,
    "backup_id": "20240813T205658"
  }
}

Diff as JSON

8a9,10
>     "children_backup_ids": null,
>     "cluster_size": null,
26a29
>     "data_checksums": null,
36a40
>     "parent_backup_id": null,
40a45
>     "summarize_wal": null,
barthisrael commented 1 month ago

So, it seems we are not breaking backward compatibility with anything. I only have two comments about the above output:

Thoughts @gcalacoci @andremagui ?

barthisrael commented 1 month ago

Backups taken with dev

Postgres method -- full

Show as console

Backup 20240814T015504:
  Backup Name            : dev_postgres_full
  Server Name            : postgres
  System Id              : 7402632902805533789
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data
  Estimated Cluster Size : 22.4 MiB

  Server information:
    Checksums            : off
    WAL summarizer       : on

  Base backup information:
    Backup Method        : postgres
    Backup Type          : full
    Backup Size          : 22.3 MiB (54.3 MiB with WALs)
    WAL Size             : 32.0 MiB
    Timeline             : 1
    Begin WAL            : 00000001000000000000002D
    End WAL              : 00000001000000000000002E
    WAL number           : 2
    Begin time           : 2024-08-14 01:55:04.975021+02:00
    End time             : 2024-08-14 01:55:06.455466+02:00
    Copy time            : 1 second
    Estimated throughput : 15.3 MiB/s
    Begin Offset         : 40
    End Offset           : 0
    Begin LSN            : 0/2D000028
    End LSN              : 0/2E000000

  WAL information:
    No of files          : 2
    Disk usage           : 32.0 MiB
    WAL rate             : 577.74/hour
    Last available       : 000000010000000000000030

  Catalog information:
    Retention Policy     : not enforced
    Previous Backup      : - (this is the oldest base backup)
    Next Backup          : 20240814T015515
    Children Backup(s)   : 20240814T015515

Show as JSON

{
  "postgres": {
    "backup_id": "20240814T015504",
    "backup_name": "dev_postgres_full",
    "backup_type": "full",
    "base_backup_information": {
      "backup_method": "postgres",
      "backup_size": "22.3 MiB",
      "backup_size_bytes": 23391484,
      "backup_size_with_wals": "54.3 MiB",
      "backup_size_with_wals_bytes": 56945916,
      "begin_lsn": "0/2D000028",
      "begin_offset": 40,
      "begin_time": "2024-08-14 01:55:04.975021+02:00",
      "begin_time_timestamp": "1723593304",
      "begin_wal": "00000001000000000000002D",
      "copy_time": "1 second",
      "copy_time_seconds": 1.461041,
      "end_lsn": "0/2E000000",
      "end_offset": 0,
      "end_time": "2024-08-14 01:55:06.455466+02:00",
      "end_time_timestamp": "1723593306",
      "end_wal": "00000001000000000000002E",
      "throughput": "15.3 MiB/s",
      "throughput_bytes": 16010148,
      "timeline": 1,
      "wal_num": 2,
      "wal_size": "32.0 MiB",
      "wal_size_bytes": 33554432
    },
    "catalog_information": {
      "children_backup_ids": [
        "20240814T015515"
      ],
      "next_backup": "20240814T015515",
      "previous_backup": "- (this is the oldest base backup)",
      "retention_policy": "not enforced"
    },
    "cluster_size": "22.4 MiB",
    "cluster_size_bytes": 23530875,
    "pgdata_directory": "/var/lib/pgsql/17/data",
    "postgresql_version": 170000,
    "server_information": {
      "data_checksums": "off",
      "summarize_wal": "on"
    },
    "status": "DONE",
    "system_id": "7402632902805533789",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "32.0 MiB",
      "disk_usage_bytes": 33554432,
      "last_available": "000000010000000000000030",
      "no_of_files": 2,
      "timelines": [],
      "wal_rate": "577.74/hour",
      "wal_rate_per_second": 0.16048252073519007
    }
  }
}

Postgres method -- incremental

Show as console

Backup 20240814T015515:
  Backup Name            : dev_postgres_incremental
  Server Name            : postgres
  System Id              : 7402632902805533789
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data
  Estimated Cluster Size : 22.4 MiB

  Server information:
    Checksums            : off
    WAL summarizer       : on

  Base backup information:
    Backup Method        : postgres
    Backup Type          : incremental
    Backup Size          : 3.0 MiB (35.0 MiB with WALs)
    WAL Size             : 32.0 MiB
    Resource savings     : 19.5 MiB (86.80%)
    Timeline             : 1
    Begin WAL            : 00000001000000000000002F
    End WAL              : 000000010000000000000030
    WAL number           : 2
    Begin time           : 2024-08-14 01:55:15.555038+02:00
    End time             : 2024-08-14 01:55:17.038603+02:00
    Copy time            : 1 second
    Estimated throughput : 2.0 MiB/s
    Begin Offset         : 40
    End Offset           : 0
    Begin LSN            : 0/2F000028
    End LSN              : 0/30000000

  WAL information:
    No of files          : 1
    Disk usage           : 16.0 MiB
    WAL rate             : 639.45/hour
    Last available       : 000000010000000000000031

  Catalog information:
    Retention Policy     : not enforced
    Previous Backup      : 20240814T015504
    Next Backup          : - (this is the latest base backup)
    Root Backup          : 20240814T015504
    Parent Backup        : 20240814T015504
    Backup chain size    : 2

Show as JSON

{
  "postgres": {
    "backup_id": "20240814T015515",
    "backup_name": "dev_postgres_incremental",
    "backup_type": "incremental",
    "base_backup_information": {
      "backup_method": "postgres",
      "backup_size": "3.0 MiB",
      "backup_size_bytes": 3106676,
      "backup_size_with_wals": "35.0 MiB",
      "backup_size_with_wals_bytes": 36661108,
      "begin_lsn": "0/2F000028",
      "begin_offset": 40,
      "begin_time": "2024-08-14 01:55:15.555038+02:00",
      "begin_time_timestamp": "1723593315",
      "begin_wal": "00000001000000000000002F",
      "copy_time": "1 second",
      "copy_time_seconds": 1.463778,
      "end_lsn": "0/30000000",
      "end_offset": 0,
      "end_time": "2024-08-14 01:55:17.038603+02:00",
      "end_time_timestamp": "1723593317",
      "end_wal": "000000010000000000000030",
      "resource_savings": "19.5 MiB",
      "resource_savings_bytes": 20424199,
      "resource_savings_percentage": "86.80%",
      "throughput": "2.0 MiB/s",
      "throughput_bytes": 2122368,
      "timeline": 1,
      "wal_num": 2,
      "wal_size": "32.0 MiB",
      "wal_size_bytes": 33554432
    },
    "catalog_information": {
      "chain_size": 2,
      "children_backup_ids": null,
      "next_backup": "- (this is the latest base backup)",
      "parent_backup_id": "20240814T015504",
      "previous_backup": "20240814T015504",
      "retention_policy": "not enforced",
      "root_backup_id": "20240814T015504"
    },
    "cluster_size": "22.4 MiB",
    "cluster_size_bytes": 23530875,
    "pgdata_directory": "/var/lib/pgsql/17/data",
    "postgresql_version": 170000,
    "server_information": {
      "data_checksums": "off",
      "summarize_wal": "on"
    },
    "status": "DONE",
    "system_id": "7402632902805533789",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "16.0 MiB",
      "disk_usage_bytes": 16777216,
      "last_available": "000000010000000000000031",
      "no_of_files": 1,
      "timelines": [],
      "wal_rate": "639.45/hour",
      "wal_rate_per_second": 0.17762601279509127
    }
  }
}

Rsync method

Show as console

Backup 20240814T015530:
  Backup Name            : dev_rsync
  Server Name            : rsync
  System Id              : 7402632902805533789
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /var/lib/pgsql/17/data
  Estimated Cluster Size : 22.4 MiB

  Server information:
    Checksums            : off
    WAL summarizer       : on

  Base backup information:
    Backup Method        : rsync-concurrent
    Backup Size          : 22.1 MiB (38.1 MiB with WALs)
    WAL Size             : 16.0 MiB
    Resource savings     : 0 B (0.00%)
    Timeline             : 1
    Begin WAL            : 000000010000000000000031
    End WAL              : 000000010000000000000031
    WAL number           : 1
    Begin time           : 2024-08-14 01:55:30.424998+02:00
    End time             : 2024-08-14 01:55:33.010967+02:00
    Copy time            : 1 second
    Estimated throughput : 12.7 MiB/s
    Begin Offset         : 40
    End Offset           : 288
    Begin LSN            : 0/31000028
    End LSN              : 0/31000120

  WAL information:
    No of files          : 0
    Disk usage           : 0 B
    Last available       : 000000010000000000000031

  Catalog information:
    Retention Policy     : not enforced
    Previous Backup      : - (this is the oldest base backup)
    Next Backup          : - (this is the latest base backup)

Show as JSON

{
  "rsync": {
    "backup_id": "20240814T015530",
    "backup_name": "dev_rsync",
    "backup_type": "rsync",
    "base_backup_information": {
      "backup_method": "rsync-concurrent",
      "backup_size": "22.1 MiB",
      "backup_size_bytes": 23225468,
      "backup_size_with_wals": "38.1 MiB",
      "backup_size_with_wals_bytes": 40002684,
      "begin_lsn": "0/31000028",
      "begin_offset": 40,
      "begin_time": "2024-08-14 01:55:30.424998+02:00",
      "begin_time_timestamp": "1723593330",
      "begin_wal": "000000010000000000000031",
      "copy_time": "1 second",
      "copy_time_seconds": 1.746225,
      "end_lsn": "0/31000120",
      "end_offset": 288,
      "end_time": "2024-08-14 01:55:33.010967+02:00",
      "end_time_timestamp": "1723593333",
      "end_wal": "000000010000000000000031",
      "resource_savings": "0 B",
      "resource_savings_bytes": 0,
      "resource_savings_percentage": "0.00%",
      "throughput": "12.7 MiB/s",
      "throughput_bytes": 13300386,
      "timeline": 1,
      "wal_num": 1,
      "wal_size": "16.0 MiB",
      "wal_size_bytes": 16777216
    },
    "catalog_information": {
      "next_backup": "- (this is the latest base backup)",
      "previous_backup": "- (this is the oldest base backup)",
      "retention_policy": "not enforced"
    },
    "cluster_size": "22.4 MiB",
    "cluster_size_bytes": 23530875,
    "pgdata_directory": "/var/lib/pgsql/17/data",
    "postgresql_version": 170000,
    "server_information": {
      "data_checksums": "off",
      "summarize_wal": "on"
    },
    "status": "DONE",
    "system_id": "7402632902805533789",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "0 B",
      "disk_usage_bytes": 0,
      "last_available": "000000010000000000000031",
      "no_of_files": 0,
      "timelines": [],
      "wal_rate": 0,
      "wal_rate_per_second": 0
    }
  }
}

Snapshot method

Show as console

Backup 20240814T015546:
  Backup Name            : dev_snapshot
  Server Name            : snapshot
  System Id              : 7402620047885836080
  Status                 : DONE
  PostgreSQL Version     : 160004
  PGDATA directory       : /opt/postgres/data
  Estimated Cluster Size : 22.7 MiB

  Server information:
    Checksums            : on

  Snapshot information:
    provider             : aws
    account_id           : 714574844897
    region               : sa-east-1

    device_name          : /dev/sdf
    snapshot_id          : snap-0d08260a1b1ddd3e3
    snapshot_name        : Barman_AWS:1:/dev/sdf-20240814t015546
    Mount point          : /opt/postgres
    Mount options        : rw,noatime,seclabel

  Base backup information:
    Backup Method        : snapshot-concurrent
    Backup Size          : 1.0 KiB (16.0 MiB with WALs)
    WAL Size             : 16.0 MiB
    Resource savings     : 0 B (0.00%)
    Timeline             : 1
    Begin WAL            : 000000010000000000000014
    End WAL              : 000000010000000000000014
    WAL number           : 1
    Begin time           : 2024-08-13 23:55:47.457466+00:00
    End time             : 2024-08-13 23:56:34.741301+00:00
    Copy time            : 46 seconds
    Estimated throughput : 22 B/s
    Begin Offset         : 40
    End Offset           : 312
    Begin LSN            : 0/14000028
    End LSN              : 0/14000138

  WAL information:
    No of files          : 0
    Disk usage           : 0 B
    Last available       : 000000010000000000000014

  Catalog information:
    Retention Policy     : not enforced
    Previous Backup      : - (this is the oldest base backup)
    Next Backup          : - (this is the latest base backup)

Show as JSON

{
  "snapshot": {
    "backup_id": "20240814T015546",
    "backup_name": "dev_snapshot",
    "backup_type": "rsync",
    "base_backup_information": {
      "backup_method": "snapshot-concurrent",
      "backup_size": "1.0 KiB",
      "backup_size_bytes": 1073,
      "backup_size_with_wals": "16.0 MiB",
      "backup_size_with_wals_bytes": 16778289,
      "begin_lsn": "0/14000028",
      "begin_offset": 40,
      "begin_time": "2024-08-13 23:55:47.457466+00:00",
      "begin_time_timestamp": "1723593347",
      "begin_wal": "000000010000000000000014",
      "copy_time": "46 seconds",
      "copy_time_seconds": 46.922532,
      "end_lsn": "0/14000138",
      "end_offset": 312,
      "end_time": "2024-08-13 23:56:34.741301+00:00",
      "end_time_timestamp": "1723593394",
      "end_wal": "000000010000000000000014",
      "resource_savings": "0 B",
      "resource_savings_bytes": 0,
      "resource_savings_percentage": "0.00%",
      "throughput": "22 B/s",
      "throughput_bytes": 22,
      "timeline": 1,
      "wal_num": 1,
      "wal_size": "16.0 MiB",
      "wal_size_bytes": 16777216
    },
    "catalog_information": {
      "next_backup": "- (this is the latest base backup)",
      "previous_backup": "- (this is the oldest base backup)",
      "retention_policy": "not enforced"
    },
    "cluster_size": "22.7 MiB",
    "cluster_size_bytes": 23812069,
    "pgdata_directory": "/opt/postgres/data",
    "postgresql_version": 160004,
    "server_information": {
      "data_checksums": "on",
      "summarize_wal": null
    },
    "snapshots_info": {
      "provider": "aws",
      "provider_info": {
        "account_id": "714574844897",
        "region": "sa-east-1"
      },
      "snapshots": [
        {
          "mount": {
            "mount_options": "rw,noatime,seclabel",
            "mount_point": "/opt/postgres"
          },
          "provider": {
            "device_name": "/dev/sdf",
            "snapshot_id": "snap-0d08260a1b1ddd3e3",
            "snapshot_name": "Barman_AWS:1:/dev/sdf-20240814t015546"
          }
        }
      ]
    },
    "status": "DONE",
    "system_id": "7402620047885836080",
    "tablespaces": [],
    "wal_information": {
      "compression_ratio": 0,
      "disk_usage": "0 B",
      "disk_usage_bytes": 0,
      "last_available": "000000010000000000000014",
      "no_of_files": 0,
      "timelines": [],
      "wal_rate": 0,
      "wal_rate_per_second": 0
    }
  }
}

Cloud method

Show as console

Backup 20240814T015916:
  Backup Name            : dev_cloud
  Server Name            : cloud
  System Id              : 7402692746266015596
  Status                 : DONE
  PostgreSQL Version     : 170000
  PGDATA directory       : /home/vagrant/pgdata-for-cloud
  Estimated Cluster Size : 22.4 MiB

  Server information:
    Checksums            : off
    WAL summarizer       : off

  Base backup information:
    Timeline             : 1
    Begin WAL            : 000000010000000000000006
    End WAL              : 000000010000000000000006
    Begin time           : 2024-08-14 01:59:17.048460+02:00
    End time             : 2024-08-14 01:59:18.660859+02:00
    Copy time            : less than one second
    Begin Offset         : 40
    End Offset           : 344
    Begin LSN            : 0/6000028
    End LSN              : 0/6000158

Show as JSON

{
  "cloud": {
    "backup_label": "'START WAL LOCATION: 0/6000028 (file 000000010000000000000006)\\nCHECKPOINT LOCATION: 0/6000080\\nBACKUP METHOD: streamed\\nBACKUP FROM: primary\\nSTART TIME: 2024-08-14 01:59:17 CEST\\nLABEL: Barman backup cloud 20240814T015916\\nSTART TIMELINE: 1\\n'",
    "backup_name": "dev_cloud",
    "begin_offset": 40,
    "begin_time": "Wed Aug 14 01:59:17 2024",
    "begin_wal": "000000010000000000000006",
    "begin_xlog": "0/6000028",
    "children_backup_ids": null,
    "cluster_size": 23530869,
    "compression": null,
    "config_file": "/home/vagrant/pgdata-for-cloud/postgresql.conf",
    "copy_stats": {
      "total_time": 1.835681,
      "number_of_workers": 2,
      "analysis_time": 0,
      "analysis_time_per_item": {
        "data": 0
      },
      "copy_time_per_item": {
        "data": 0.587703
      },
      "serialized_copy_time_per_item": {
        "data": 0.472939
      },
      "copy_time": 0.587703,
      "serialized_copy_time": 0.472939
    },
    "data_checksums": "off",
    "deduplicated_size": null,
    "end_offset": 344,
    "end_time": "Wed Aug 14 01:59:18 2024",
    "end_wal": "000000010000000000000006",
    "end_xlog": "0/6000158",
    "error": null,
    "hba_file": "/home/vagrant/pgdata-for-cloud/pg_hba.conf",
    "ident_file": "/home/vagrant/pgdata-for-cloud/pg_ident.conf",
    "included_files": null,
    "mode": null,
    "parent_backup_id": null,
    "pgdata": "/home/vagrant/pgdata-for-cloud",
    "server_name": "cloud",
    "size": null,
    "status": "DONE",
    "summarize_wal": "off",
    "systemid": "7402692746266015596",
    "tablespaces": null,
    "timeline": 1,
    "version": 170000,
    "xlog_segment_size": 16777216,
    "backup_id": "20240814T015916"
  }
}
barthisrael commented 1 month ago

I think the backups taken with the dev branch also look good. I have nothing to add to the comments from a previous reply.

gcalacoci commented 1 month ago

So, it seems we are not breaking backward compatibility with anything. I only have two comments about the above output:

  • I guess we removed number_of_workers from the JSON output due to an oversight;
  • I think it doesn't make sense to output Resource savings for snapshot backups. That is useful for Postgres incremental backups, and for rsync backups only.

Thoughts @gcalacoci @andremagui ?

IMHO json output should not hide stuff. but just none it. so if resource saving is none, we output none... if you get what I mean. I guess that the number of worker stuff is an oversight and yeah for the above reasoning, it should be there @barthisrael @andremagui

barthisrael commented 1 month ago

IMHO json output should not hide stuff. but just none it. so if resource saving is none, we output none... if you get what I mean.

@gcalacoci I meant to take out Resource savings from the barman -f console show-backup for snapshot-based backups. Are we talking about the same or about different things here? 😆

andremagui commented 1 month ago

@gcalacoci @barthisrael I will add number_of_workers to the json output. Just one thing though, each different type of backup is hiding something in the JSON output, such as some incremental stuff (parent_backup id, ...) is hidden in rsync and full postgres backup. What would be the reasoning here? Show we show the same exactly output for any backup type with None for the fields that are just not there or hide specifically?

andremagui commented 1 month ago

@barthisrael and I discussed briefly about the comments above and we are going to proceed with what we have now, insert number_of_workers back in the json output, remove resource_savings from snapshots and show only specific fields for each type of backup.

barthisrael commented 1 month ago

Just complementing, in the future we might rethink this JSON output so we get a unified layout that is able to accommodate things independently of the underlying method.

The current implementation on Barman 3.10.1 already plays with this "hiding mechanism" -- although it plays with that hiding sections instead of fields.

We might review this in the future and refactor this output in a proper way.

andremagui commented 1 month ago

Just as NOTE: The last snapshot test from @barthisrael have the backup_type="rsync" but this is being fixed in the PR #984 . This happens because the property was just taking into consideration "rsync", "full" and "incremental".

barthisrael commented 1 month ago

It seems unit tests got broken.

barthisrael commented 1 month ago

I ran a new series of tests. I took backups with all methods using 3.10.1 and attempted to show them using this dev branch, both in console and json formats. Neither of them are throwing exceptions

barthisrael commented 1 month ago

The number_of_workers field is back:

$ barman -f json show-backup postgres dev_postgres_full | jq | grep number
      "number_of_workers": 1,

$ barman -f json show-backup postgres dev_postgres_incremental | jq | grep number
      "number_of_workers": 1,

$ barman -f json show-backup rsync dev_rsync | jq | grep number
      "number_of_workers": 1,

$ barman -f json show-backup snapshot dev_snapshot | jq | grep number
      "number_of_workers": 1,

$ barman-cloud-backup-show --endpoint-url=http://127.0.0.1:4566 -P barman-cloud s3://barman-s3-test cloud 20240814T182535 --format json | jq | grep number
      "number_of_workers": 2,
barthisrael commented 1 month ago

This field still shows for snapshot on console format, though:

$ barman show-backup snapshot dev_snapshot | grep Resource
    Resource savings     : 0 B (0.00%)
barthisrael commented 1 month ago

I can see you introduced this code:

        if backup_type and backup_type in {"rsync", "incremental"}:
            dedupe_output = "{} ({})".format(
                pretty_size(est_dedup_size),
                "{percent:.2%}".format(percent=deduplication_ratio),
            )
            output_fun(nested_row.format("Resource savings", dedupe_output))

Once #984 is merged, this Resource savings will be gone from the output of barman show-backup for snapshot-base backups.

andremagui commented 1 month ago

Here are the tests made with PostreSQL version 16 for the show-backup command and cloud-backup-show. Apparently nothing is broken the compatibility.

backup method: Postgres 

Backup 20240814T053912:
  Server Name            : pg16
  System Id              : 7403077509859742493
  Status                 : DONE
  PostgreSQL Version     : 160004
  PGDATA directory       : /var/lib/pgsql/16/data
  Estimated Cluster Size : 22.5 MiB

  Server information:
    Checksums            : off

  Base backup information:
    Backup Method        : postgres
    Backup Type          : full
    Backup Size          : 22.3 MiB (22.4 MiB with WALs)
    WAL Size             : 16.1 KiB
    Timeline             : 1
    Begin WAL            : 00000001000000000000001C
    End WAL              : 00000001000000000000001C
    WAL number           : 1
    WAL compression ratio: 99.90%
    Begin time           : 2024-08-14 18:28:55.647737-03:00
    End time             : 2024-08-14 18:28:57.931030-03:00
    Copy time            : 2 seconds
    Estimated throughput : 9.9 MiB/s
    Begin Offset         : 40
    End Offset           : 0
    Begin LSN            : 0/1C000028
    End LSN              : 0/1D000000

  WAL information:
    No of files          : 0
    Disk usage           : 0 B
    Last available       : 00000001000000000000001C

  Catalog information:
    Retention Policy     : not enforced
    Previous Backup      : 20240814T053817
    Next Backup          : - (this is the latest base backup)

backup method: rsync 

Backup 20240814T053953:
  Server Name            : pg16
  System Id              : 7403077509859742493
  Status                 : DONE
  PostgreSQL Version     : 160004
  PGDATA directory       : /var/lib/pgsql/16/data
  Estimated Cluster Size : 22.5 MiB

  Server information:
    Checksums            : off

  Base backup information:
    Backup Method        : rsync-concurrent
    Backup Size          : 22.2 MiB (22.2 MiB with WALs)
    WAL Size             : 16.1 KiB
    Resource savings     : 0 B (0.00%)
    Timeline             : 1
    Begin WAL            : 00000001000000000000001E
    End WAL              : 00000001000000000000001E
    WAL number           : 1
    WAL compression ratio: 99.90%
    Begin time           : 2024-08-14 18:29:37.227037-03:00
    End time             : 2024-08-14 18:29:38.262406-03:00
    Copy time            : less than one second
    Estimated throughput : 42.3 MiB/s (2 jobs)
    Begin Offset         : 40
    End Offset           : 256
    Begin LSN            : 0/1E000028
    End LSN              : 0/1E000100

  WAL information:
    No of files          : 0
    Disk usage           : 0 B
    Last available       : 00000001000000000000001E

  Catalog information:
    Retention Policy     : not enforced
    Previous Backup      : 20240814T053912
    Next Backup          : - (this is the latest base backup)

backup method: rsync with resource_savings

Backup 20240814T102800:
  Server Name            : pg16
  System Id              : 7403077509859742493
  Status                 : DONE
  PostgreSQL Version     : 160004
  PGDATA directory       : /var/lib/pgsql/16/data
  Estimated Cluster Size : 22.5 MiB

  Server information:
    Checksums            : off

  Base backup information:
    Backup Method        : rsync-concurrent
    Backup Size          : 8.9 KiB (25.0 KiB with WALs)
    WAL Size             : 16.1 KiB
    Resource savings     : 22.5 MiB (99.96%)
    Timeline             : 1
    Begin WAL            : 000000010000000000000020
    End WAL              : 000000010000000000000020
    WAL number           : 1
    WAL compression ratio: 99.90%
    Begin time           : 2024-08-14 23:17:44.306969-03:00
    End time             : 2024-08-14 23:17:45.192832-03:00
    Copy time            : less than one second
    Estimated throughput : 20.0 KiB/s (2 jobs)
    Begin Offset         : 40
    End Offset           : 256
    Begin LSN            : 0/20000028
    End LSN              : 0/20000100

  WAL information:
    No of files          : 0
    Disk usage           : 0 B
    Last available       : 000000010000000000000020

  Catalog information:
    Retention Policy     : not enforced
    Previous Backup      : 20240814T053953
    Next Backup          : - (this is the latest base backup)

backup method: cloud using barman-cloud-backup-show command

Backup 20240814T095400:
  Server Name            : cloud
  System Id              : 7402970942348298308
  Status                 : DONE
  PostgreSQL Version     : 160004
  PGDATA directory       : /var/lib/pgsql/16/data

  Base backup information:
    Timeline             : 1
    Begin WAL            : 000000010000000000000005
    End WAL              : 000000010000000000000005
    Begin time           : 2024-08-14 09:54:00.286426-03:00
    End time             : 2024-08-14 09:54:00.778026-03:00
    Copy time            : 1 second
    Begin Offset         : 40
    End Offset           : 256
    Begin LSN            : 0/5000028
    End LSN              : 0/5000100
andremagui commented 1 month ago

Now in JSON format backup method: postgres

{
    "pg16": {
        "backup_id": "20240814T053912",
        "backup_type": "full",
        "base_backup_information": {
            "backup_method": "postgres",
            "backup_size": "22.3 MiB",
            "backup_size_bytes": 23422065,
            "backup_size_with_wals": "22.4 MiB",
            "backup_size_with_wals_bytes": 23438526,
            "begin_lsn": "0/1C000028",
            "begin_offset": 40,
            "begin_time": "2024-08-14 18:28:55.647737-03:00",
            "begin_time_timestamp": "1723670935",
            "begin_wal": "00000001000000000000001C",
            "copy_time": "2 seconds",
            "copy_time_seconds": 2.264037,
            "end_lsn": "0/1D000000",
            "end_offset": 0,
            "end_time": "2024-08-14 18:28:57.931030-03:00",
            "end_time_timestamp": "1723670937",
            "end_wal": "00000001000000000000001C",
            "number_of_workers": 1,
            "throughput": "9.9 MiB/s",
            "throughput_bytes": 10345266,
            "timeline": 1,
            "wal_compression_ratio": "99.90%",
            "wal_num": 1,
            "wal_size": "16.1 KiB",
            "wal_size_bytes": 16461
        },
        "catalog_information": {
            "children_backup_ids": null,
            "next_backup": "20240814T053953",
            "previous_backup": "20240814T053817",
            "retention_policy": "not enforced"
        },
        "cluster_size": "22.5 MiB",
        "cluster_size_bytes": 23558123,
        "pgdata_directory": "/var/lib/pgsql/16/data",
        "postgresql_version": 160004,
        "server_information": {
            "data_checksums": "off",
            "summarize_wal": null
        },
        "status": "DONE",
        "system_id": "7403077509859742493",
        "tablespaces": [],
        "wal_information": {
            "compression_ratio": "99.90%",
            "disk_usage": "32.2 KiB",
            "disk_usage_bytes": 32940,
            "last_available": "00000001000000000000001E",
            "no_of_files": 2,
            "timelines": [],
            "wal_rate": "254.77/hour",
            "wal_rate_per_second": 0.07077036610380032
        }
    }
}

backup method: rsync

{
    "pg16": {
        "backup_id": "20240814T053953",
        "backup_type": "rsync",
        "base_backup_information": {
            "backup_method": "rsync-concurrent",
            "backup_size": "22.2 MiB",
            "backup_size_bytes": 23248992,
            "backup_size_with_wals": "22.2 MiB",
            "backup_size_with_wals_bytes": 23265451,
            "begin_lsn": "0/1E000028",
            "begin_offset": 40,
            "begin_time": "2024-08-14 18:29:37.227037-03:00",
            "begin_time_timestamp": "1723670977",
            "begin_wal": "00000001000000000000001E",
            "copy_time": "less than one second",
            "copy_time_seconds": 0.524093,
            "end_lsn": "0/1E000100",
            "end_offset": 256,
            "end_time": "2024-08-14 18:29:38.262406-03:00",
            "end_time_timestamp": "1723670978",
            "end_wal": "00000001000000000000001E",
            "number_of_workers": 2,
            "resource_savings": "0 B",
            "resource_savings_bytes": 0,
            "resource_savings_percentage": "0.00%",
            "throughput": "42.3 MiB/s",
            "throughput_bytes": 44360432,
            "timeline": 1,
            "wal_compression_ratio": "99.90%",
            "wal_num": 1,
            "wal_size": "16.1 KiB",
            "wal_size_bytes": 16459
        },
        "catalog_information": {
            "next_backup": "20240814T102800",
            "previous_backup": "20240814T053912",
            "retention_policy": "not enforced"
        },
        "cluster_size": "22.5 MiB",
        "cluster_size_bytes": 23558123,
        "pgdata_directory": "/var/lib/pgsql/16/data",
        "postgresql_version": 160004,
        "server_information": {
            "data_checksums": "off",
            "summarize_wal": null
        },
        "status": "DONE",
        "system_id": "7403077509859742493",
        "tablespaces": [],
        "wal_information": {
            "compression_ratio": "99.90%",
            "disk_usage": "32.2 KiB",
            "disk_usage_bytes": 33024,
            "last_available": "000000010000000000000020",
            "no_of_files": 2,
            "timelines": [],
            "wal_rate": "0.62/hour",
            "wal_rate_per_second": 0.0001735415152380219
        }
    }
}

backup method: rsync with resource_savings

{
    "pg16": {
        "backup_id": "20240814T102800",
        "backup_type": "rsync",
        "base_backup_information": {
            "backup_method": "rsync-concurrent",
            "backup_size": "8.9 KiB",
            "backup_size_bytes": 9158,
            "backup_size_with_wals": "25.0 KiB",
            "backup_size_with_wals_bytes": 25620,
            "begin_lsn": "0/20000028",
            "begin_offset": 40,
            "begin_time": "2024-08-14 23:17:44.306969-03:00",
            "begin_time_timestamp": "1723688264",
            "begin_wal": "000000010000000000000020",
            "copy_time": "less than one second",
            "copy_time_seconds": 0.446857,
            "end_lsn": "0/20000100",
            "end_offset": 256,
            "end_time": "2024-08-14 23:17:45.192832-03:00",
            "end_time_timestamp": "1723688265",
            "end_wal": "000000010000000000000020",
            "number_of_workers": 2,
            "resource_savings": "22.5 MiB",
            "resource_savings_bytes": 23548843,
            "resource_savings_percentage": "99.96%",
            "throughput": "20.0 KiB/s",
            "throughput_bytes": 20494,
            "timeline": 1,
            "wal_compression_ratio": "99.90%",
            "wal_num": 1,
            "wal_size": "16.1 KiB",
            "wal_size_bytes": 16462
        },
        "catalog_information": {
            "next_backup": "- (this is the latest base backup)",
            "previous_backup": "20240814T053953",
            "retention_policy": "not enforced"
        },
        "cluster_size": "22.5 MiB",
        "cluster_size_bytes": 23558123,
        "pgdata_directory": "/var/lib/pgsql/16/data",
        "postgresql_version": 160004,
        "server_information": {
            "data_checksums": "off",
            "summarize_wal": null
        },
        "status": "DONE",
        "system_id": "7403077509859742493",
        "tablespaces": [],
        "wal_information": {
            "compression_ratio": 0,
            "disk_usage": "0 B",
            "disk_usage_bytes": 0,
            "last_available": "000000010000000000000020",
            "no_of_files": 0,
            "timelines": [],
            "wal_rate": 0,
            "wal_rate_per_second": 0
        }
    }
}

backup method: cloud

{
    "cloud": {
        "backup_label": "'START WAL LOCATION: 0/5000028 (file 000000010000000000000005)\\nCHECKPOINT LOCATION: 0/5000060\\nBACKUP METHOD: streamed\\nBACKUP FROM: primary\\nSTART TIME: 2024-08-14 09:54:00 -03\\nLABEL: Barman backup cloud 20240814T095400\\nSTART TIMELINE: 1\\n'",
        "begin_offset": 40,
        "begin_time": "Wed Aug 14 09:54:00 2024",
        "begin_wal": "000000010000000000000005",
        "begin_xlog": "0/5000028",
        "children_backup_ids": null,
        "cluster_size": null,
        "compression": null,
        "config_file": "/var/lib/pgsql/16/data/postgresql.conf",
        "copy_stats": {
            "total_time": 1.610534,
            "number_of_workers": 2,
            "analysis_time": 0,
            "analysis_time_per_item": {
                "data": 0
            },
            "copy_time_per_item": {
                "data": 1.222805
            },
            "serialized_copy_time_per_item": {
                "data": 0.315637
            },
            "copy_time": 1.222805,
            "serialized_copy_time": 0.315637
        },
        "data_checksums": null,
        "deduplicated_size": null,
        "end_offset": 256,
        "end_time": "Wed Aug 14 09:54:00 2024",
        "end_wal": "000000010000000000000005",
        "end_xlog": "0/5000100",
        "error": null,
        "hba_file": "/var/lib/pgsql/16/data/pg_hba.conf",
        "ident_file": "/var/lib/pgsql/16/data/pg_ident.conf",
        "included_files": null,
        "mode": null,
        "parent_backup_id": null,
        "pgdata": "/var/lib/pgsql/16/data",
        "server_name": "cloud",
        "size": null,
        "status": "DONE",
        "summarize_wal": null,
        "systemid": "7402970942348298308",
        "tablespaces": null,
        "timeline": 1,
        "version": 160004,
        "xlog_segment_size": 16777216,
        "backup_id": "20240814T095400"
    }
}