migrate dev/prod to aws

[hongsudt]

Before migration:

• [ ] setup aws restic backup
• [ ] Upgrade OS to fedora38.
• [ ] create aws recipe: server spec: 2 core, 8GB of RAM with 1) 20GB EBS for OS, 2) 110GB EBS for "data" virtual disk
  • [ ] set up lvm with "data" volume group
  • /var/lib/pgsql: 30GB (currently use 5GB on www, 6.5GB on dev/staging),
  • /var/www/: 50GB (currently use 9GB + 4GB for legacy)
  • /scratch: 20GB (currently use 3x mmCIF file with largest size of 2GB so far -- 6GB)
  • /home: 10GB (currently use 3.4GB)

# current system disk usage 
df -h
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/data-pgsql     28G  4.5G   24G  17% /var/lib/pgsql
/dev/mapper/data-home     7.9G  3.4G  4.5G  43% /home
128.9.34.103:/export/pdb  3.2T  4.3G  3.2T   1% /var/www
tmpfs                     795M  4.0K  795M   1% /run/user/1010

instance names: dev-aws.pdb-dev.org, data-aws.pdb-dev.org

Before migration: data-aws.pdb-dev.org

• [ ] make sure ermrest db, hatrac db, and hatrac are backedup in aws
• [ ] Restore ermrest from backup
• [ ] Restore hatrac from backup
• [ ] Make sure everything works properly
• [ ] Change data.pdb-dev.org to point to the new server

[svoinea]

# current files sizes

select 'entry' as "Table", 'Image_File_Name' as "File", count(*) as "Number of Records", to_char(min("Image_File_Bytes"), 'FM9,999,999,999,999') as "Minimum Size", to_char(max("Image_File_Bytes"), 'FM9,999,999,999,999') as "Maximum Size", to_char(avg("Image_File_Bytes")::integer, 'FM9,999,999,999,999') as "Average Size", to_char(sum("Image_File_Bytes"), 'FM9,999,999,999') as "Total Size" from "PDB"."entry" where "Image_File_Bytes" is not null and "Image_File_Bytes" != 0
union
select 'entry' as "Table", 'mmCIF_File_Name' as "File", count(*) as "Number of Records", to_char(min("mmCIF_File_Bytes"), 'FM9,999,999,999,999') as "Minimum Size", to_char(max("mmCIF_File_Bytes"), 'FM9,999,999,999,999') as "Maximum Size", to_char(avg("mmCIF_File_Bytes")::integer, 'FM9,999,999,999,999') as "Average Size", to_char(sum("mmCIF_File_Bytes"), 'FM9,999,999,999') as "Total Size" from "PDB"."entry" where "mmCIF_File_Bytes" is not null and "mmCIF_File_Bytes" != 0
union
select 'Entry_mmCIF_File' as "Table", 'File_Name' as "File", count(*) as "Number of Records", to_char(min("File_Bytes"), 'FM9,999,999,999,999') as "Minimum Size", to_char(max("File_Bytes"), 'FM9,999,999,999,999') as "Maximum Size", to_char(avg("File_Bytes")::integer, 'FM9,999,999,999,999') as "Average Size", to_char(sum("File_Bytes"), 'FM9,999,999,999') as "Total Size" from "PDB"."Entry_mmCIF_File" where "File_Bytes" is not null and "File_Bytes" != 0
union
select 'Entry_Error_File' as "Table", 'File_Name' as "File", count(*) as "Number of Records", to_char(min("File_Bytes"), 'FM9,999,999,999,999') as "Minimum Size", to_char(max("File_Bytes"), 'FM9,999,999,999,999') as "Maximum Size", to_char(avg("File_Bytes")::integer, 'FM9,999,999,999,999') as "Average Size", to_char(sum("File_Bytes"), 'FM9,999,999,999') as "Total Size" from "PDB"."Entry_Error_File" where "File_Bytes" is not null and "File_Bytes" != 0
union
select 'ihm_starting_model_details' as "Table", 'mmCIF_File_Name' as "File", count(*) as "Number of Records", to_char(min("mmCIF_File_Bytes"), 'FM9,999,999,999,999') as "Minimum Size", to_char(max("mmCIF_File_Bytes"), 'FM9,999,999,999,999') as "Maximum Size", to_char(avg("mmCIF_File_Bytes")::integer, 'FM9,999,999,999,999') as "Average Size", to_char(sum("mmCIF_File_Bytes"), 'FM9,999,999,999') as "Total Size" from "PDB"."ihm_starting_model_details" where "mmCIF_File_Bytes" is not null and "mmCIF_File_Bytes" != 0
union
select 'Entry_Related_File' as "Table", 'File_Name' as "File", count(*) as "Number of Records", to_char(min("File_Bytes"), 'FM9,999,999,999,999') as "Minimum Size", to_char(max("File_Bytes"), 'FM9,999,999,999,999') as "Maximum Size", to_char(avg("File_Bytes")::integer, 'FM9,999,999,999,999') as "Average Size", to_char(sum("File_Bytes"), 'FM9,999,999,999') as "Total Size" from "PDB"."Entry_Related_File" where "File_Bytes" is not null and "File_Bytes" != 0
union
select 'Entry_Related_File_Templates' as "Table", 'File_Name' as "File", count(*) as "Number of Records", to_char(min("File_Bytes"), 'FM9,999,999,999,999') as "Minimum Size", to_char(max("File_Bytes"), 'FM9,999,999,999,999') as "Maximum Size", to_char(avg("File_Bytes")::integer, 'FM9,999,999,999,999') as "Average Size", to_char(sum("File_Bytes"), 'FM9,999,999,999') as "Total Size" from "PDB"."Entry_Related_File_Templates" where "File_Bytes" is not null and "File_Bytes" != 0
order by "Table", "Total Size" desc;

            Table             |      File       | Number of Records | Minimum Size | Maximum Size | Average Size |  Total Size   
------------------------------+-----------------+-------------------+--------------+--------------+--------------+---------------
 entry                        | mmCIF_File_Name |               204 | 11,777       | 379,873,143  | 10,579,985   | 2,158,316,875
 entry                        | Image_File_Name |               114 | 81,496       | 8,420,936    | 1,404,531    | 160,116,580
 Entry_Error_File             | File_Name       |                15 | 142          | 63,873,106   | 6,856,996    | 102,854,938
 Entry_mmCIF_File             | File_Name       |               110 | 969          | 167,109,524  | 9,713,293    | 1,068,462,217
 Entry_Related_File           | File_Name       |                40 | 84           | 75,454       | 9,364        | 374,572
 Entry_Related_File_Templates | File_Name       |                22 | 26           | 282          | 132          | 2,900
 ihm_starting_model_details   | mmCIF_File_Name |               230 | 22,109       | 16,428,568   | 959,572      | 220,701,642
(7 rows)

[svoinea]

Total files sizes: 3.7G.

[hongsudt]

From experiences shared by Synapse: 3:08 PM FWIW, I have ongoing experiments looking successful with my migration of synapse to AWS. I've got the synapse staging server there for a few days. Our stack of deriva services and postgres all seem to work fine on the Fedora 38 aarch64 (ARM cpu) OS on the t4g (ARM graviton) instance type at EC2.

• to populate content in the new VM, I restored content from restic backups rather than via rsync etc.
• I figured out how to configure reverse-proxy on the old ISI host so all traffic goes to AWS even while I wait for action to update DNS for my synapse-staging.isrd.isi.edu hostname
• I'm also using the hatrac "overlay" solution to cheaply reuse the shared production S3 content while new stuff goes in a local filesystem
• I used hatrac's alternate naming scheme for s3 so I could rclone my production content from our ISRD NAS volume into the production s3 bucket

The reverse-proxy setup to move traffic ahead of DNS changes is a little tricky. The main issue is that webauthn embeds hostnames inside responses and everything needs to line up to the same origin for a webauthn+Globus login to succeed. A typical reverse-proxy setup won't work (where the old host knows the new host by a different name), because it only rewrites a subset of response fields.

• set the new VM to have the same hostname so our recipes that are sensitive to it will setup the same apache config file contents
• restore the same TLS certificate files onto the new host so it can be authenticated for the normal public name(s) of the existing server
• set /etc/hosts in the OLD host to point the public hostname(s) to the NEW host's IP address(es)
• set reverse proxy rules in OLD apache config to route to the public hostname(s) again, which will then lookup that IP address in /etc/hosts

With this kind of setup, web clients can use the public hostname(s) in URLs and end up talking to the new server, with the old server acting as just a proxy/tunnel in the short term. Once DNS records are updated, they would then talk directly to the new server.And before setting up the reverse proxy, you can also edit your local client /etc/hosts to allow browser-based testing of the new server via the public name. Of course, that will only apply to your one system where the browser runs. The reverse proxy is to help all other clients move as well, if you don't want to wait for the DNS change delays... (edited)

[hongsudt]

@svoinea the hatrac path with some characters such as spaces will not work with the flask version. Have you fixed these paths?

[brindakv]

@hongsudt Please review this issue.