Create backup on RCSB cloud east

[brindakv]

Create ermrest and hatrac backup on RCSB east cloud.

@svoinea will provide a tarball.

Create an account for @svoinea / proxy-ihm-account on RCSB east cloud.

Create a dropbox folder with access to @svoinea.

[brindakv]

Communicating with RCSB IT admins.

[brindakv]

@hongsudt is looking at backing up on s3 bucket.

[svoinea]

A daily backup for the database and hatrac was enabled for the dev/staging and production on apache-pdbdev-deposit-east.wwpdb.org.

As staging and dev are sharing the same VM, a single backup was scheduled which goes on the apache-pdbdev-deposit-east.wwpdb.org:/mnt/vdb1/PDB-Dev-deriva-DHS/staging directory. Its database directory contains both catalogs:

• _ermrest_catalog_99.sql.gz for dev
• _ermrest_catalog_50.sql.gz for staging

A pair of ed25519 keys was generated in this respect for the user pdbihm.

[brindakv]

Additional notes:

• The cron job that carries out the daily backup from USC to RCSB cloud happens at 3:00 AM PT / 6:00 AM ET. The daily backup of data in RCSB cloud happens around 4:30 AM ET. The backup of the DHS on the IQB hard drive will be delayed by a day.

• apache-pdbdev-deposit-east.wwpdb.org:/mnt/vdb1/PDB-Dev-deriva-DHS/dev is empty because apache-pdbdev-deposit-east.wwpdb.org:/mnt/vdb1/PDB-Dev-deriva-DHS/staging includes both staging and dev instances.

• database corresponds to postgresql dump (not just ermRest).

• @svoinea will review the current practice at ISI-USC and update the namespace accordingly (e.g., psql). Suggestion:

  • apache-pdbdev-deposit-east.wwpdb.org:/mnt/vdb1/PDB-Dev-deriva-DHS/staging --> apache-pdbdev-deposit-east.wwpdb.org:/mnt/vdb1/PDB-Dev-deriva-DHS/dev.pdb-dev.org

  • apache-pdbdev-deposit-east.wwpdb.org:/mnt/vdb1/PDB-Dev-deriva-DHS/production --> apache-pdbdev-deposit-east.wwpdb.org:/mnt/vdb1/PDB-Dev-deriva-DHS/data.pdb-dev.org

[brindakv]

@brindakv will find out about wwPDB practice regarding the frequency of backing up data in the OneDep system and @svoinea will find out about current practice at ISI. For example, single day (overwritten every day), multiple days (days 0, 1, 2, etc. for seven days), weekly (overwritten every week), etc.

[svoinea]

The /mnt/vdb1/PDB-Dev-deriva-DHS directory on apache-pdbdev-deposit-east.wwpdb.org was restructured such it contains now the two PDB hostnames:

• dev.pdb-dev.org
• data.pdb-dev.org

Each of them contains the directory structure of the original source:

• for the database: /var/lib/pgsql/14/backups
• for the hatrac: /var/www/hatrac

[brindakv]

@svoinea will add home/pdb-ihm to the backups.

@brindakv will review the backups.

For later: Try restore using backups.

[svoinea]

Added in the backups the /home/pdbihm with the following sub-directories used by the backend script:

• config
• cpp-dict-pack
• make-mmCIF
• py-rcsb_db

[brindakv]

@brindakv will check with IQB IT team regarding backups on IQB storage.

@hongsudt and @brindakv will look at using restic for backups.

[brindakv]

Checked with the IQB IT team and got confirmation that the data is backed up on the local IQB storage.