search

HDFGroup / hsds

Cloud-native, service based access to HDF data
https://www.hdfgroup.org/solutions/hdf-kita/
Apache License 2.0
128 stars 52 forks source link

Enable direct access support #94

Closed jreadey closed 2 years ago

jreadey commented 3 years ago

Enable HSDS to be used directly by applications (without a server needing to be setup beforehand) as described in this design doc: https://github.com/HDFGroup/hsds/blob/master/docs/design/direct_access/direct_access.md.

jreadey commented 2 years ago

This is working now in master (hsds & h5pyd). Enable by setting the h5pyd endpoint to "local".

Example:

$ hsinfo --endpoint local
server name: Direct Connect (HSDS)
server state: READY
endpoint: local
username: john
password: *******
home: /home/john/
server version: 0.7.0beta
node count: 6
up: 4 sec
h5pyd version: 0.9.0

There's a startup penalty of ~2 seconds, but the subprocesses will stay open until the file handle is closed in h5pyd. Performance is similar to using HSDS on the local host.

jreadey commented 2 years ago

This is supported in the v0.7beta11 release.