Document the process to alter the primary node for a series of collections from S3 to Minio – begin with Dash > Dryad

[elopatin-uc3]

Summary

A series of 44 collections that are owned by CDL and campuses continue to use S3 as their primary storage node. The total storage capacity used by these collections as of March 2020 is 5.6TB. For the purpose of cost savings and simplification, the majority of these collections should be reconfigured to use Minio as a primary storage node, with Glacier and Wasabi as secondary nodes. In some cases, ownership changes will also be needed in the Inventory database (e.g. Dash > Dryad).

High-priority Collections

The first collections to migrate will be the existing Dash collections. They also require an ownership change. All Dash collections should migrate from primary node 5001, to primary node 3041.

Collection ARK	Old owner	New owner	mnemonic
ark:/13030/m5q82t8x	UCB Lib (ark:/13030/j2x634h1)	Dryad (ark:/13030/j2br86wx)	ucb_dash
ark:/13030/m50s13nf	LBNL (ark:/13030/j21n79mc)	Dryad (ark:/13030/j2br86wx)	lbnl_dash
ark:/13030/m5v45qp2	UCD Lib (ark:/13030/j2sf2761)	Dryad (ark:/13030/j2br86wx)	ucd_lib_dash
ark:/13030/m5fr19qh	UCI Lib (ark:/13030/j2np19wm)	Dryad (ark:/13030/j2br86wx)	uci_dash
ark:/13030/m5tm8r6v	UCLA Lib (ark:/13030/j28910x4)	Dryad (ark:/13030/j2br86wx)	ucla_dash
ark:/13030/m5b00k0h	UCM Lib (ark:/13030/j2d7929g)	Dryad (ark:/13030/j2br86wx)	ucm_dash
ark:/13030/m5jt2v2m	UCR Lib (ark:/13030/j28g850g)	Dryad (ark:/13030/j2br86wx)	ucr_lib_dash
ark:/13030/m5xs78k8	UCSB (ark:/13030/j2pc294b)	Dryad (ark:/13030/j2br86wx)	ucsb_dash
ark:/13030/m5t16hvv	UCSC Lib (ark:/13030/j2w6645c)	Dryad (ark:/13030/j2br86wx)	ucsc_dash
ark:/13030/m52j8gvj	UCSF Lib (ark:/13030/j2rf56vz)	Dryad (ark:/13030/j2br86wx)	ucsf_lib_datashare

Collection ARK	Old owner	New owner	mnemonic
ark:/13030/m5v13jxb	UC3 (ark:/13030/j20v83bf)	Dryad (ark:/13030/j2br86wx)	ucop_dash
ark:/13030/m5dz54vh	UC Press (ark:/13030/j2rx9247)	Dryad (ark:/13030/j2br86wx)	ucpress_dash

Tasks

• [ ] Document the overall steps required for reconfiguring these collections, leveraging methodology used during the Minio migration where possible.
• [ ] Spin off new tickets for associated work, starting with Dash collection migration.
• [x] Confirm that UCOP and UC Press Dash collections should migrate to Dryad – CONFIRMED, these should be treated the same as all other Dash collections

[elopatin-uc3]

From Scott: " these are the non-Dryad collections we have configured in our application from the old Dash."

dataone.yml: endpoint: "http://mrtsword.cdlib.org:39001/mrtsword/collection/dataone_dash" lbnl.yml: endpoint: "http://mrtsword.cdlib.org:39001/mrtsword/collection/lbnl_dash" ucb.yml: endpoint: "http://mrtsword.cdlib.org:39001/mrtsword/collection/ucb_dash" ucd.yml: endpoint: "http://mrtsword.cdlib.org:39001/mrtsword/collection/ucd_lib_dash" uci.yml: endpoint: "http://mrtsword.cdlib.org:39001/mrtsword/collection/uci_dash" ucla.yml: endpoint: "http://mrtsword.cdlib.org:39001/mrtsword/collection/ucla_dash" ucm.yml: endpoint: "http://mrtsword.cdlib.org:39001/mrtsword/collection/ucm_dash" ucop.yml: endpoint: "http://mrtsword.cdlib.org:39001/mrtsword/collection/ucop_dash" (not enabled) ucpress.yml: endpoint: "http://mrtsword.cdlib.org:39001/mrtsword/collection/ucpress_dash" ucr.yml: endpoint: "http://mrtsword.cdlib.org:39001/mrtsword/collection/ucr_lib_dash" ucsb.yml: endpoint: "http://mrtsword.cdlib.org:39001/mrtsword/collection/ucsb_dash" ucsc.yml: endpoint: "http://mrtsword.cdlib.org:39001/mrtsword/collection/ucsc_dash" ucsf.yml: endpoint: "http://mrtsword.cdlib.org:39001/mrtsword/collection/ucsf_lib_datashare"

[elopatin-uc3]

@dloy I've updated this ticket with ARK information for all DASH collections. Once a procedure is documented, these are the collections to start replication to Dryad first, followed by flipping the primary node.

[elopatin-uc3]

@dloy Scott confirmed with Daniella that the UCOP and UC Press Dash collections should be treated the same as all of the others; no differences in terms of replication or migration.

[elopatin-uc3]

Related to #482

[elopatin-uc3]

Moving this off the project board and back to the Merritt Futures board, for when we begin dealing with the other collections that have their primary object copies in S3.