search

cianfrocco-lab / COSMIC-CryoEM-Gateway

Web platform for analyzing cryo-EM data on Comet cluster at San Diego Supercomputer Center
http://cosmic-cryoem.org
GNU General Public License v3.0
9 stars 1 forks source link

New storage #219

Closed monaw closed 4 years ago

monaw commented 4 years ago

From Gavin Billfar @ SDSC

As the universal scale storage service continues to expand, changes are needed to support the growth. We are implementing a more robust erasure coding scheme to provide better data protection. Unfortunately, downtime will be needed during the cutover and a new CNAME record will be introduced. I’ve already migrated the data to the new erasure coding stripe and there is a job running continuously to keep the new target synchronized. The cutover should only take a few minutes. We can follow this process:

  1. I’ll set the existing NFS export read-only (qstorage.sdsc.edu:/cosmic2)
  2. I’ll perform a final sync to ensure the source and target are mirrored
  3. I’ll set the new exports read-writable (qs-cosmic.sdsc.edu:/cosmic2)
  4. The NFS client admins will need to swap the mount. I posted the list of the clients below, I believe all are managed by RDS and DESC?

Alternatively, I can expose the new export and allow the client(s) to manage the migration. After the cutover, I plan to maintain the old export for some time before destroying the data and reclaiming the capacity. Please let me know if you have any questions or concerns with the process, and when you’d like to schedule the cutover. I realize the impact of this change and I want to do everything I can to help with the process.

monaw commented 4 years ago

also from Gavin:

The data is being replicated at the block level. All higher level file system ownership, permissions, POSIX mode bits, links, etc. will be the same. I’ve already cutover a number of other ‘qstorage’ NFS exports without issue.

I plan to set the existing storage read-only during the cutover process and maintain that data for some time – it’ll be the original/production file system. I was going to keep the original data for at least 30 days after the cutover. That said, I don’t plan to delete the existing storage without your approval.

monaw commented 4 years ago

new storage tested and working