search

sul-dlss / dor-services-app

A Rails application exposing Digital Object Registry functions as a RESTful HTTP API
https://sul-dlss.github.io/dor-services-app/
Other
3 stars 2 forks source link

[SPIKE] Parallelize migration #3680

Closed justinlittman closed 2 years ago

justinlittman commented 2 years ago

Currently, the migration script works serially. It is estimated that will require ~60 days (at approx. 1 object per second).

For this spike, experiment with parallelizing the script. The goal is to migrate faster without knocking over Fedora, so minor parallelization is probably optimal. (That is, we're going for faster but not blazing.)

There are examples of parallelization of similar in bin/validate-*.

The sdr-migration environment should be used for testing. For questions about the migration script see @ndushay .

justinlittman commented 2 years ago

Alternatively, could just split the list and run multiple instances of the migration script in screen.

justinlittman commented 2 years ago

Done.