informatics-isi-edu / hatrac

Simple object storage for collaborations
Apache License 2.0
3 stars 1 forks source link

add hatrac-async-migrate tool and supporting backend API changes #70

Open karlcz opened 10 months ago

karlcz commented 10 months ago

This tool does not assume that two hatrac servers exist, nor does it call any hatrac REST API to move data. Instead, it exports work from the service database into sqlite3 work file(s) to be processed by workers that have access to both source and destination backing stores. Progress is recorded back to the sqlite work file(s).

The existing (source) service can operate unaffected while the workers replicate bulk data to the destination store. A final merge step is performed with the service offline in order to reintegrate destination storage information back into the service database. After merging, the service should be restarted with the destination storage configuration.

The backend API enhancements allow the migration tool to supply existing object-version IDs during replication, so that the bulk storage may embed this predefined version ID in the storage name rather than generating a new random ID, where applicable.