Louisville - Fedora move and restore

[crisr15]

Summary

After exploring the modspace errors, it was determined that the Fedora used in prod is not backed by postgres. This has caused Fedora corruption as many records were added, and many of the branches have been broken. In order to resolve this we need to do a new Fedora setup.

We then want to mod the actor stack to check for thumbnails and assets before running file tasks. Then we re-build each Bulkrax:Entry after moving to an empty postgres backed fcrepo. This would then create all the metadata records with out having to redo the time consuming file transfers.

Acceptance Criteria

Code adjustments and a restore script should exist such that neither the binary files in the Fedora directory nor the derivative timestamps change when recreating the fcrepo.

Testing Instructions

Proposed Implementation Steps

1) Boot louisville hyku locally and import 5 or 6 works with the file based fcrepo via Bulkrax. 2) copy the fcrepo directory and the derivatives directory to a backup location 3) change the fcrepo to be postgres backed (and thus blank). 4) in the rails console run

e = Bulkrax::Importer.find(YOUR_IMPORTER_ID_FROM_STEP_1).entries
e[0].build

check if the derivative process reruns (file timestamps can help with this), check if the binary in fcrepo is recreated. Make code changes until both are true. (You'll have 5 shots before you have to restore the backups in step 3 and try again)

5) write a script which will requeue Bulkrax::Entry for louisville. Pay attention to relationships if they are needed. restore from step (3) and clear out fcrepo postgres table. restart fcrepo. Run your script and see that all fcrepo records are restored

6) review process with Randy so he can run it. Empasise that a backup of solr, the database and fcrepo should be taken before starting. if they are all on a VM's disk, a disk snapshot should be fine for this.

[bkiahstroud]

Results of script

Cleanup needed

Summary

• Many relationships are missing
• Many thumbnails are missing
• Some Work records appear to be missing their file attachments

Details

Missing relationships

• Bulkrax::PendingRelationship.count --> 0
• Need to:
  • Locate entries with missing relationships
  • Create corresponding Bulkrax::PendingRelationship records
  • Process all relationships in jobs

Missing thumbnails

• The Sidekiq queue has many (~750) SetDefaultParentThumbnailJobs that don't appear to be going anywhere
• The SetDefaultParentThumbnailJob appears to heavily rely on relationships existing between records
  • Need to see if SetDefaultParentThumbnailJobs succeed after fixing relationships

Missing file attachments

• Need to check if this is also resolved by fixing relationships
  • Files not appearing on Work pages could be caused by a missing relationship between the two

[bkiahstroud]

Randy was able to get this across the line via reimporting, so this is now done