CIRM service migration epic

[robes]

This is an umbrella issue for the migration of CIRM data services from their lab to our data center. They currently host their services on dinsdale.hsc.usc.edu which include virtual hosts:

• piranha for storage
• spineynorman for ermrest
• axioscan2014 (I forget the hostname) for Zeiss Zen Web server on Windows

We are migrating them to a setup in ISI's data centers that includes:

• cirm-staging for ermrest and hatrac
• ISI's new data services for bulk data

The plan is to bring up our services here and make them operational for collecting new data. CIRM's services on dinsdale will continue to run in parallel in a read-only mode for a period of time for us to migrate the bulk data onto our services. Once we complete the data migration, we can decommission spiney and piranha.

Phase I: bring up services at ISI

• [x] we should get the cirm-staging service (ermrest + hatrac) operational here at ISI
• [x] cirm-staging should be using the ISI bulk storage service
• [x] globus configuration and certs etc should all be ready to be validated
• [x] test that we can print labels
• [x] deploy IObox on CIRM's acquisition workstation
• [x] use project (cirmusc) credentials (not personal accounts) for the IObox agent
• [x] test upload of data from the IObox (at CIRM !!) to cirm-staging
• at this point we can declare that we have an operational system at ISI that is ready to begin migration phases

Phase II: migrate database and ongoing operations to ISI, done on a flag day of sorts

• [x] disable writes to spineynorman's ermrest (ie, switch off the content_write_user acl)
• [x] dump and backup the spineynorman postgres database
• [x] load the CIRM postgres database on cirm-staging
• [x] validate that it is operational
• at this point CIRM should be operationally migrated to ISI while their local services on dinsdale remain intact for read-only access and we can begin the bulk data migration to ISI

Phase IIIa: migrate the bulk data to ISI

• [x] transfer CZI files from piranha to cirm-staging (to its bulk data partition)
• [x] we may need to review and adjust file level ACLs in hatrac
• [x] in the database, rewrite any fully qualified URLs to the CZI files to point to cirm-staging's hatrac locations for the files
• [x] in the database, we can also clear out any remaining pointers (URLs) to data on piranha such as the Zoomify tiles and TIFF files that are no longer needed
• at this point CIRM's services can be decommissioned and they may recycle them for other use or leave them as backup

Phase IIIb: generate tiled images

• this phase is not strictly sequential with IIIa and can start in parallel
• [x] we may need to make some minor changes to the database schema to record the location of the (DZI) tiled images
• [x] finished and deploy script(s) for generating (DZI) tiled images from the CZI; these will be installed on cirm-staging
• [x] begin the process of tiling the images, which should also update the database with the necessary information for launching the bare Seadragon viewer; note: we are not starting with the annotation viewer because that involves further modification of the database schema which falls outside the scope of this migration
• at this point we have a fully migrated system that recreates the present capabilities of the locally deployed services

[robes]

I believe this epic is finished now.