search

avalonmediasystem / avalon

Avalon Media System – Samvera Application
http://www.avalonmediasystem.org/
Apache License 2.0
93 stars 51 forks source link

Test migration from Fedora 4 to 6 #5364

Closed cjcolvar closed 11 months ago

cjcolvar commented 1 year ago

Testing will be done in #5357 with a new empty Avalon on Fedora 6. Just as important if not more so will be to see if Avalon can work with data migrated from Fedora 4.

Done Looks Like

cjcolvar commented 1 year ago

Tried walking through the migration instructions inside the avalon_archivo-stage container:

#Install java on archivo-staging container: 
apt-get install openjdk-8-jre
#Install newer java: 
curl -O -L https://download.java.net/java/GA/jdk21/fd2272bbf8e04c3dbaee13770090416c/35/GPL/openjdk-21_linux-x64_bin.tar.gz
tar xvzf openjdk-21_linux-x64_bin.tar.gz
#Downloaded fedora import/export tool (recommended 0.3.0 version):
curl -O -L https://github.com/fcrepo-exts/fcrepo-import-export/releases/download/fcrepo-import-export-0.3.0/fcrepo-import-export-0.3.0.jar
#Run export tool: 
jdk-21/bin/java -jar fcrepo-import-export-0.3.0.jar -b --dir archivo_fcrepo4.7.5_export --user fedoraAdmin:fedoraAdmin --mode export --resource http://localhost:8080/fedora/rest --binaries --versions
#INFO 15:21:34.345 (Exporter) Running exporter...
#...
#INFO 15:23:43.351 (Exporter) Exported http://localhost:8080/fedora/rest/09/3e/dc/e3/093edce3-a094-4c46-b066-5d91fd06807f to /home/app/tmp/export/archivo_fcrepo4.7.5_export/fedora/rest/09/3e/dc/e3/093edce3-a094-4c46-b066-5d91fd06807f.ttl
#Did this actually finish?
#Download migration tool: 
curl -v -O -L https://github.com/fcrepo-exts/fcrepo-upgrade-utils/releases/download/fcrepo-upgrade-utils-6.2.0/fcrepo-upgrade-utils-6.2.0.jar
#Run migration to fedora 5.x: 
jdk-21/bin/java -jar fcrepo-upgrade-utils-6.2.0.jar --input-dir archivo_fcrepo4.7.5_export --output-dir archivo_fcrepo5_export --source-version 4.7.5 --target-version 5+
#INFO 15:22:27.786 (F47ToF5UpgradeManager) Processing directory: /home/app/tmp/export/archivo_fcrepo4.7.5_export
#java.lang.RuntimeException: java.nio.file.NoSuchFileException: archivo_fcrepo4.7.5_export/fedora/rest/af/21/ab/71/af21ab71-c690-4c3b-a608-a76dd1529044
    at org.fcrepo.upgrade.utils.F47ToF5UpgradeManager.convertAcl(F47ToF5UpgradeManager.java:372)
#...
#Run migration from fedora 5.x to 6: 
jdk-21/bin/java --add-opens java.base/java.util.concurrent=ALL-UNNAMED -jar fcrepo-upgrade-utils-6.2.0.jar --input-dir archivo_fcrepo5_export --output-dir archivo_fcrepo6_export --source-version 5+ --target-version 6+ --base-uri http://localhost:8080/fedora/rest
#INFO 15:43:18.988 (F5ToF6UpgradeManager) Starting upgrade: config=Config{sourceVersion=V_5, targetVersion=V_6, inputDir=archivo_fcrepo5_export, outputDir=archivo_fcrepo6_export, srcRdfLang=Lang:Turtle, baseUri='http://localhost:8080/fedora/rest', threads=4, digestAlgorithm='null', fedoraUser='fedoraAdmin', fedoraUserAddress='info:fedora/fedoraAdmin', forceWindowsMode=false, writeToS3=false, s3Region='null', s3Endpoint='null', s3PathStyleAccess=false, s3Bucket=null, s3Prefix=null, resourceInfoFile=null}
#...
#...
#INFO 15:43:23.722 (ResourceMigrator) Failed to migration resource info:fedora/76/53/71/40/76537140z. Rolling back...   #ERROR 15:43:23.724 (MigrateResourceTask) Failed to process ResourceInfo{parentId='info:fedora', fullId='info:fedora/76/53/71/40/76537140z', nameEncoded='76537140z', outerDirectory=archivo_fcrepo5_export/fedora/rest/76/53/71/40, innerDirectory=archivo_fcrepo5_export/fedora/rest/76/53/71/40/76537140z, type=CONTAINER}
#java.lang.IllegalStateException: Failed to identify interaction model for resource info:fedora/76/53/71/40/76537140z
#...
#INFO 15:43:28.850 (F5ToF6UpgradeManager) Upgrade complete.
cjcolvar commented 1 year ago

I investigated the error further and found that the migration tool was looking for directories that didn't exist. It appears they weren't created for ACL nodes which didn't have any children. I reran the export and migration steps but after exporting ran a set of shell commands to create directories ACL nodes:

grep -r "ns002:accessControl" archivo_fcrepo4.7.5_export_backup/ | cut -d '<' -f 2 | cut -d '>' -f 1 | sort | uniq > 4.7.5_acl_ids
cat 4.7.5_acl_ids | cut -d '/' -f 6- > 4.7.5_acl_id_paths
for path in $(cat 4.7.5_acl_id_paths)
do
  mkdir "archivo_fcrepo4.7.5_export_backup/fedora/rest/${path}"
done

This allowed the migrations to run successfully! I created an issue for this in the migration tool repo: https://github.com/fcrepo-exts/fcrepo-upgrade-utils/issues/55