test metadig-engine on k8s against a hashstore

[jeanetteclark]

Testing locally has gone well but it would be nice to test the engine against a hashstore on the dev cluster

to that end I've mounted the tdg subvolume on metadig-worker, and that subvolume was mounted on dev.nceas where there is a hashstore metacat running. See helm/metadig-worker/pv.yaml and helm/metadig-worker/pvc.yaml for details on the existing mounts.

In order to actually test though the following steps are needed:

• [ ] copy the contents of metacat/hashstore to /mnt/tdg-repos/dev via parallel Rsync
• [ ] symlink that hashstore to metacat
• [ ] update the metacat.properties store.store_path field to be var/data/respos/dev/hashstore
• [ ] deploy metadig-engine to the test cluster
• [ ] submit datasets to dev.nceas (via metacatUI or any other client)

[doulikecookiedough]

Update:

The rsync + parallel process to copy the contents of /var/metacat/hashstore to /mnt/tdg-repos/dev/metacat/hashstore has been completed.

• The re-sync process takes approximately 25 minutes to complete with just the first level folders/items in the /var/metacat/hashstore folder.
  • I am re-running the process with individual rsync commands for each file in the mean time to see if it's any faster

Next Steps:

• Sync up with Jing to coordinate metacat switchover (and symlinking new directory)

To Do List:

• [x] copy the contents of metacat/hashstore to /mnt/tdg-repos/dev via parallel Rsync
• [x] symlink that hashstore to metacat
• [x] update the metacat.properties store.store_path field to be /mnt/tdg-repos/dev/metacat/hashstore
  • Note: This does not have to be done since we created a symlink
• [ ] deploy metadig-engine to the test cluster
• [ ] submit datasets to dev.nceas (via metacatUI or any other client)

For reference:

# How to produce a text file with just the first level of hashstore folders to rsync
mok@dev:~/testing$ sudo find /var/metacat/hashstore -mindepth 1 -maxdepth 1 > mc_hs_dir_list.txt
mok@dev:~/testing$ cat mc_hs_dir_list.txt
/var/metacat/hashstore/objects
/var/metacat/hashstore/metadata
/var/metacat/hashstore/refs
/var/metacat/hashstore/hashstore.yaml

# How to use rsync with a list of folders
mok@dev:~/testing$ cat mc_hs_dir_list.txt | parallel --eta sudo rsync -aHAX {} /mnt/tdg-repos/dev/metacat/hashstore/

# First get the list of files found under `/hashstore`
mok@dev:~/testing$ sudo find /var/metacat/hashstore -type f -printf '%P\n' > mc_obj_list.txt

# How to feed a single command at a time for a file to rsync
# The /./ between `metacat` and `hashstore` instructs rsync to copie folders from hashstore (and omits the previous directories) into the desired folder
mok@dev:~/testing$ parallel --eta sudo rsync -aHAXR /var/metacat/./hashstore/{} /mnt/tdg-repos/dev/metacat :::: mc_obj_list.txt

• Note: Not defining the amount of cores will default rsync to determine its own limit (which when undefined, went to the max # of cores (ex. 44). When I added -j 30 it was limited to 30.)

[doulikecookiedough]

Metacat on dev.nceas.ucsb.edu has been moved over to write to the ceph fs mount point - a symlink has been created between /var/metacat/hashstore and /mnt/tdg-repos/dev/metacat/hashstore.

• Note: We initially ran into a read-only file system issue that was caused due to how tomcat set-up its access control rules (the actual path to write above needed to be added to its configuration settings).

rsync was re-ran and the process to sync with a list of direct subfolders after /var/metacat/hashstore was the fastest. I tested with feeding rsync individual commands (ex. via :::: list_of_files.txt) but this seemed to be very slow. The re-sync process took approximately 5 minutes.

[doulikecookiedough]

Current Status:

It appears the 'Assessment Reports' (Metadig) for datasets at dev.nceas.ucsb.edu is not working as expected:

• Example: https://dev.nceas.ucsb.edu/quality/urn%3Auuid%3A313d899d-dc77-435d-9638-abd09faf7143
• Shows the error message:

  There was an error generating the assessment report.
  The Assessment Server reported this error:
  Unable to run quality suite for pid urn:uuid:313d899d-dc77-435d-9638-abd09faf7143, suite FAIR-suite-0.4.0
  Failed : HTTP error code : 403 Return to the dataset

Next Steps:

1) Restoring expected Metadig functionality @ dev.nceas.ucsb.edu

• The dev cluster's metadig-controller, metadig-scorer and metadig-scheduler are all on image v3.0.2 - except for metadig-worker which is using the feature-hashstore-support image. Before attempting to deploy the feature-hashstore-support image to the scorer, scheduler and controller per Jeanette's instructions, I will restore metadig-worker to using image v3.0.2 to try and resolve the issue on the test site.

2) Obtaining the last missing feature-hashstore-support image for metadig-controller

• metadig-controller also does not have a feature-hashstore-support image. This will require the execution of mvn publish while on the correct branch for the metadig-engine. I likely do not have appropriate permissions and will seek assistance from Jing to move forward here.

3) Deploying feature-hashstore-support for Metadig in full on the dev cluster

• Once all the images are available, I will deploy the feature-hashstore-support as per such after updating the image.tag in the respective values.yaml files (four total for each Metadig-engine piece)
• Below are the commands for quick reference:

    helm upgrade metadig-scheduler ./metadig-scheduler --namespace metadig --set image.pullPolicy=Always --recreate-pods=true --set k8s.cluster=dev
    helm upgrade metadig-scorer ./metadig-scorer --namespace metadig --set image.pullPolicy=Always --recreate-pods=true --set k8s.cluster=dev
    helm upgrade metadig-worker ./metadig-worker --namespace metadig --set image.pullPolicy=Always --set replicaCount=1 --recreate-pods=true --set k8s.cluster=dev
    helm upgrade metadig-controller ./metadig-controller --namespace metadig --set image.pullPolicy=Always --recreate-pods=true --set k8s.cluster=dev

To Do List & Follow-up Questions

• [x] copy the contents of metacat/hashstore to /mnt/tdg-repos/dev via parallel Rsync
• [x] symlink that hashstore to metacat
• [x] update the metacat.properties store.store_path field to be /mnt/tdg-repos/dev/metacat/hashstore
  • Note: This does not have to be done since we created a symlink
• [ ] Fix broken Assessment reports on dev.nceas.ucsb.edu
• [ ] Deploy metadig-engine to the test cluster
  • Once deployed, determine how to check that a data quality check can be requested/is working as expected (a way to directly communicate with Metadig?)
• [ ] Submit datasets to dev.nceas (via metacatUI or any other client)
  • Running the quality checks at scale (many objects) to make sure the system is performing without errors