Move arcticdata.io (Production) to Kubernetes

artntek commented 3 months ago

Similar to #1932;

checklist:

[x] Work with @nickatnceas to copy production data for testing:
- [x] Time how long it takes to...
  - [x] Copy the production postgres data (arcticdata.io:/var/lib/postgresql) to the PROD ceph volume at /mnt/ceph/repos/arctic/postgresql (treat it like a hot backup).
    - NOTE: we do not need the /var/lib/postgresql/10 directory
  - [x] copy the following subset of production data from arcticdata.io:/var/metacat to the PROD ceph volume at /mnt/ceph/repos/arctic/metacat:
```
# /var/metacat/...
16K     ./certs
63T     ./data
8.0K        ./dataone
3.9G        ./documents
0           ./inline-data
500K        ./logs
```
    - Actual Times taken for /var/metacat/data:
      - initial rsync
        root@arctica:/var/metacat# time rsync -aHAX --delete /var/metacat/data/ /mnt/pdg/repos/arctic/metacat/data/
      real 14286m43.628s user 1131m15.740s sys 3907m38.871s
      
      -> 9.92 days
```
- subsequent repeat rsync
```shell
      brooke@arctica:~$ time sudo rsync -rltDHX  /var/metacat/data/ /mnt/pdg/repos/arctic/metacat/data/
      [sudo] password for brooke:
      
      real    4m19.047s
      user    0m15.747s
      sys 0m34.912s
```

Follow the Quick Reference: Metacat K8s Installation Steps. Supplementary TODOs below...

Persistent Volumes

[x] Set up a PV to point to PROD cephfs .../repos/arctic/metacat for metacat
[x] Set up a PV to point to PROD cephfs .../repos/arctic/postgres for postgres
[x] Create a PVC for Postgresql; see prod_cluster/metacatarctic/pvc--metacatarctic-postgres.yaml
[x] "csi-cephfs-sc-ephemeral" storageClass missing. Ask @nickatnceas to add, like he did for dev cluster:
```
storageclass.storage.k8s.io "csi-cephfs-sc-ephemeral" not found
```

MetacatUI setup

[x] Copy config (tokens) from adc server

Metacat Config

[x] Add values.yaml overrides for non-default 2.19 settings (diff arcticdata.io $TOMCAT_HOME/webapps/metacat/WEB-INF/metacat.properties with default metacat.properties from 2.19 release)
[x] Add values.yaml overrides for newly-introduced 3.0 settings (diff default metacat.properties from 3.0.0 release with default metacat.properties from 2.19 release)
[x] Compare with test.arcticdata.io values overrides as a sanity check

First Deployment

[x] Complete steps in "First Install - IMPORTANT IF MOVING DATA FROM AN EXISTING LEGACY DEPLOYMENT" BEFORE first startup!
[x] solr pods not starting. root cause from logs:
```
$ kc logs pod/metacatarctic-solr-1
/scripts/setup.sh: line 8: /opt/bitnami/scripts/solr/entrypoint.sh: Permission denied
```
SOLVED - was overriding extraVolumes values, and the override didn't include the permissions line
[x] https://arctic-prod.test.dataone.org/catalog/ (trailing slash) works, but https://arctic-prod.test.dataone.org/catalog gives a 404 (nginx)
- See https://github.com/kubernetes/ingress-nginx/issues/646
- See https://stackoverflow.com/questions/71424259/why-url-re-writing-is-not-working-when-i-do-not-use-slash-at-the-end
```
- path: /link1(/|$)
```
[x] ensure all data and documents files are group writeable (otherwise, hashstore upgrader can't create hard links):
```
sudo find /mnt/ceph/repos/arctic/metacat/data/ -type f ! -perm -g=w -exec chmod g+w {} +
```

[x] chown -R 59997:59997 the ceph dir corresponding to /var/metacat, and update values.yaml to use this uid:gid

brooke@datateam:/mnt/ceph/repos/arctic$ time sudo chown -R 59997:59997 metacat

real    4m7.026s
user    0m0.004s
sys 0m0.027s

[x] Hostname aliases and rewrite rules
- Figure out how to do these with ingress; see all-sites-enabled.conf. Lots of complexity - eg http://aoncadis.org aliased to adc.io, and site conf has RewriteMaps each having >3700 entries.
- EXPLANATION: aoncadis.org was the predecessor to the ADC site. These rewrite rules map existing, old dataset urls to their new locations on ADC - so these rewrites need to be maintained somewhere
- [x] Leave all the redirects/other sites on the current Apache host, and move only arcticdata.io.

ATTENTION: Still To Do Before Final Deployment

[x] Time hashstore conversion
[ ] Time reindex-all
[ ] MetacatUI + WordPress setup. How do we host it and link to k8s metacat?
- [ ] ACTION: use a wordpress image/bitnami chart, deployed separately from the metacat helm chart
[x] ACTION: Ask @nickatnceas for help with letsencrypt certs - do we need to remove arcticdata.io from wildcard cert on arctica? NOTE: we still need subdomain certs there (ie status.adc, beta.adc).
[ ] Skip 3.0.0 and deploy 3.1.0, but only after it's been running on less-trafficked hosts for a while. See proposed release plan in Issue #1984.

Testing - see Matt's comment below

mbjones commented 1 month ago

For the Testing section, here's a quick rundown:

Get the R package `dataone` installed

Download and install R (required) and RStudio (nice but optional)
Checkout rdataone and (probably) switch to the develop branch, depending on what you need to test
open rdataone/dataone.RProj in RStudio
run install.packages(c('remotes', 'devtools'))
run devtools::load_all() to load the current dataone library code for testing
run remotes::install_deps() to install all of the package dependencies

Run the tests against standard nodes

login to https://dev.nceas.ucsb.edu and copy your token for R from the web UI; paste the token options command into the R console and run it
run devtools::test() to run the original tests against standard nodes

To run tests against a different node

Modify the nodes to be tested in the test framework file: https://github.com/DataONEorg/rdataone/blob/develop/tests/testthat/helper-base.R
- For example, change line 21 d1cTestKNB <- D1Client("STAGING", "urn:node:mnTestKNB") to use a different node for CRUD tests
ensure you are logged in as described above
run devtools::test() as described above

artntek commented 1 week ago

hashstore conversion notes

See #1964

First conversion (with errors) took almost exactly 48 hours

artntek commented 3 days ago

11/19/24: Second conversion (comprising only the failed objects from last time) took 42 minutes (Douglas Adams would approve)

see https://github.com/NCEAS/metacat/issues/1964#issuecomment-2484552430 for error analysis

artntek commented 3 days ago

11/19/24: Did another rsync and clean hashstore conversion

brooke@arctica:~$ time sudo rsync -aHAX --delete /var/lib/postgresql/ /mnt/ceph/repos/$NAME/postgresql/
real    60m57.679s
user    1m5.106s
sys 4m33.743s

time sudo rsync -rltDHX --stats --human-readable /var/metacat/data/ /mnt/ceph/repos/$NAME/metacat/data/
real    29m29.133s
user    1m56.979s
sys 6m45.844s

time sudo rsync -rltDHX --stats --human-readable /var/metacat/dataone/ /mnt/ceph/repos/$NAME/metacat/dataone/
real    0m10.742s
user    0m0.037s
sys 0m0.018s

time sudo rsync -rltDHX --stats --human-readable /var/metacat/documents/ /mnt/ceph/repos/$NAME/metacat/documents/
real    0m16.327s
user    0m0.490s
sys 0m1.014s

time sudo rsync -rltDHX --stats --human-readable /var/metacat/logs/ /mnt/ceph/repos/$NAME/metacat/logs/
real    0m0.101s
user    0m0.025s
sys 0m0.016s

hashstore conversion started: Wed Nov 20 22:55:15 UTC 2024

NCEAS / metacat