Upgrade from Fedora 4.2 to 4.5

[phebal]

This will involve some data migration so it'll require a backup and needs to be done after hours. Maybe coordinated with our admins?

[phebal]

1. Put Fedora/Sufia down.
2. Take a backup of fedora data
3. Upgrade to Fedora 4.3.0
4. Migrate the changed file metadata There are some utils to help with this, but they don't seems work and I'm not yet sure why: https://github.com/fcrepo4-exts/fcrepo4-upgrade-utils
5. Bring up Sufia locally, don't allow public access and test
6. If everything is ok, allow public access else restore from backup and go back to brooding
7. Upgrade to Fedora 4.5 - no migration needed it seems

[phebal]

All this needs to be done on staging first.

[phebal]

Can migrate the data successfully but running into: 403 STATUS: Objects cannot be created under pairtree nodes error when trying to create objects in the upgraded instance of Fedora.

Will try on a copy of production data to see if the same thing happens.

[phebal]

Metadata migration of the production data took 1m57.635s on my laptop, so it should take 5-10 minutes in production. Not bad.

[phebal]

Plan of migration that actually works, verified on a copy of production data:

Put Fedora/Sufia down

sudo puppet agent --disable
sudo systemctl stop httpd
sudo systemctl stop galter-resque
sudo systemctl stop galter-jetty
# Double check line numbers in iptables
sudo iptables -L -n --line-numbers
#FIXME - verify line number in the chain and replace the IP address
#sudo iptables -R INPUT 11 -p tcp -s <VPN IP ADDRESS> -m multiport --ports 80,443 -j ACCEPT

Take a @backup of fedora data

Full backup will be taken by the system administrators in the form of vm snapshot a day earlier.

Local backup: cp -a /var/www/apps/galter_hydra_jetty/shared{,.bak.4.2}

Migrate the changed file metadata with https://github.com/fcrepo4-exts/fcrepo4-upgrade-utils

Verify first: java -Dfcrepo.home=/var/www/apps/galter_hydra_jetty/shared/fcrepo4-data -jar /home/deploy/fcrepo-upgrade-utils-4.3.0.jar dryrun

If OK: `java -Dfcrepo.home=/var/www/apps/galter_hydra_jetty/shared/fcrepo4-data -jar /home/deploy/fcrepo-upgrade-utils-4.3.0.jar

**Deploy tag galter_1.6 hydra-jetty to production

Bring up Sufia locally, don't allow public access, and test

sudo systemctl start galter-jetty
sudo systemctl start httpd
sudo systemctl start galter-resque

Test the migration and Sufia operation

• [x] Verify that you can upload a file and create a collection
• [x] Verify that IIF viewer works properly
• [x] Verify that permissions objects work as intended with cancancan
• [x] Verify GenericFile, Pages and Collection counts (TBD before migration)
• [x] Run the PNB filename fixing script to see if it works like intended
• [x] Look at the objects in Fedora to make sure they look right

Allow public access

sudo systemctl reload iptables
sudo puppet agent --enable

In case of problems If something is not right restore from local backup, or if something's really really not right, ask admins to restore the vm snapshot.

[vioil]

when are you going to do this? don't do it on a friday :smile:

[phebal]

It's going to have to be not during the work hours. I think I'll wake up at 4 on Tuesday or Thursday and will do it then. I'll prepare everything ahead of time so I don't have to think much during the upgrade...

[phebal]

I'm planning to do it on Thursday, starting at 3am. I notified our sysadmins and asked them to coordinate.

[phebal]

Ticket in Footprints: 445741

[phebal]

Done, almost aborted the attempt because of a stray lock file in Fedora's data directory and java's terrible, unreadable logs but in the end it worked. We have a backup in case of any problems.