Enable direct access to DataONE data store

NCEAS / metadig-engine

MetaDig Engine: multi-dialect metadata assessment engine

7 stars 5 forks source link

Enable direct access to DataONE data store #223

Open gothub opened 5 years ago

gothub commented 5 years ago

Currently the quality engine has to acquire metadata and data via the DataONE API, which involves transferring all data from an MN or the CN each time a quality suite/pid report is created.

Enable/configure a mechanism that will allow for direct access to backend data store (metacat, GMN?) via remotely shared filesystems so that data can be efficiently shared between the quality engine on k8s an DataONE CNs, MNs.

gothub commented 5 years ago

@mbjones @csjx how can this be done?

potentially use sshfs or similiar to mount the remote disk with metacat filestore
use the Java DataONE d1_libclient_java to convert a pid to a filename

What are other options? What else needs to be done?

mbjones commented 5 years ago

@gothub let's discuss and spend a little time at the whiteboard for this one.