Checklist: Metacat K8s Upgrade Steps

1. Values: Edit your existing values override file

[ ] Remove any uid or gid overrides, since we're now adopting the defaults of 59996 for postgres and 59997 for metacat
[ ] Temporarily disable probes until hashstore conversion is done

2. Prep Before Upgrading

[ ] set storage.hashstore.disableConversion: true, so the hashstore converter won't run yet

[ ] In the metacat database, verify that all the systemmetadata.checksum_algorithm entries are on the list of supported algorithms (NOTE: syntax matters! E.g. sha-1 is OK, but sha1 isn't):

kubectl exec ${RELEASE_NAME}-postgresql-0 -- bash -c "psql -U metacat << EOF
  SELECT DISTINCT checksum_algorithm FROM systemmetadata WHERE checksum_algorithm NOT IN
        ('MD2','MD5','SHA-1','SHA-256','SHA-384','SHA-512','SHA-512/224','SHA-512/256');
EOF"

# then manually update each to the correct syntax; e.g:
kubectl exec ${RELEASE_NAME}-postgresql-0 -- bash -c "psql -U metacat << EOF
  UPDATE systemmetadata SET checksum_algorithm='SHA-1' WHERE checksum_algorithm='SHA1';
EOF"
# ...etc

3. Upgrading - NOTE THERE WILL BE DOWNTIME!

= = = = = = = = Downtime starts here = = = = = = = =

[ ] Change ownership ON CEPHFS as follows:

## postgres (59996:59996) in postgresql data directory
sudo chown -R 59996:59996 /mnt/ceph/repos/REPO-NAME/postgresql

## tomcat (59997:59997) in metacat directory
sudo chown -R 59997:59997 data dataone documents logs

[ ] ...then ensure all metacat data and documents files have g+rw permissions, otherwise, hashstore converter can't create hard links:
```
sudo chmod -R g+rw data documents dataone
```
[ ] helm upgrade, debug any startup and configuration issues, and allow hashstore upgrade to finish.

NOTE: while hashstore conversion is still in progress, it is expected for metacatUI to display Oops! It looks like there was a problem retrieving your search results., and for /metacat/d1/mn/v2/ api calls to display Metacat has not been configured

See Tips, below for how to detect when hashstore conversion finishes

= = = = = = = = Downtime ends here = = = = = = = =

[ ] When hashstore conversion has finished, re-enable probes and helm upgrade to apply changes

NCEAS / metacat

Upgrade k8s test.arcticdata.io from 3.0.0 to 3.1.0 and run tests #2029