Open rdlrt opened 7 months ago
In addition to the above mentioned items, separate to that but also connected to dump/restore. I just wanted to add that I have found that from a performance perspective, its better to compress outside of pg_dump (-Z0). Additionally, we can leverage our multi-core cpu (-j and pigz -p) to speed up the process. pigz being an external application might be a bit controversial as it needs to be added to nix flow but parallel gzip compression does help greatly when the horsepower exists to utilize it.
Some example code for how I have dumped and restored with great performance gains:
# dump
db="explorer";
tmp_dir="/tmp/dump_${db}";
threads=$(( nproc - 1 ));
pg_dump -d ${db} -Fd -j ${threads} -Z0 -f "${tmp_dir}";
tar -cf - "${tmp_dir}/" | pigz -p ${threads} > "${tmp_dir}.tar.gz";
# restore
db="explorer";
tmp_dir="/tmp/dump_${db}";
threads=$(( nproc - 1 ));
pigz -p ${threads} -dc /tmp/dump_cexplorer.tar.gz | tar -C "${tmp_dir}" --strip-components 1 -xf -;
pg_restore -j ${threads} -Fd -O -d ${db} "${tmp_dir}";
Take it for what it is... I wanted to put it out there if found useful to possibly integrate (or parts of it) in the flow for dbsync dump/restore.
Versions The
db-sync
version (egcardano-db-sync --version
): 13.2.0.1 PostgreSQL version: 16.2Build/Install Method The method you use to build or install
cardano-db-sync
: Release binariesRun method The method you used to run
cardano-db-sync
(eg Nix/Docker/systemd/none): systemdAdditional context The file
scripts/postgresql-setup.sh
is currently lagging in compatibility:exit-on-error
flag against pg_restore, we now need to specify--no-owner
and--role=<role>
which wasnt required before, thisrole
may need to be devised from PGPASSFILE (for socket, that would mean current user) OR it could be accepted as command line argument. Other alternative is adapt pg_dump to use no-owner too.disable-ledger
flag, we do not expect to have lstate files, but absence of lstate file breaks the script. Allow this flag to be optional.public
schema is filtered to be added to snapshot. Thus, dbsync users can make use of different schemas to store their addendum on the top of dbsync. This would also help leverage internal compression options from postgres