HSDS is a web service that implements a REST-based web service for HDF5 data stores. Data can be stored in either a POSIX files system, or using object-based storage such as AWS S3, Azure Blob Storage, or MinIO. HSDS can be run a single machine with or without Docker or on a cluster using Kubernetes (or AKS on Microsoft Azure).
Launch a Codespaces environment by clicking the banner "Open in GitHub Codespaces". Once the codespace is ready, type:
python testall.py
in the terminal window to run the test suite.
Make sure you have Python 3 and Pip installed, then:
$ ./build.sh --no-lint --no-docker
from source tree OR install from pypi: $ pip install hsds
$ mkdir ~/hsds_data
$ hsds --root_dir ~/hsds_data
$ export USER_NAME=$USER
$ export USER_PASSWORD=$USER
$ export ADMIN_USERNAME=$USER
$ export ADMIN_PASSWORD=$USER
$ python testall.py --skip_unit
To shut down the server, and the server is not running in Docker, just control-C.
If using docker, run: $ ./stopall.sh
Note: passwords can (and should for production use) be modified by changing values in hsds/admin/config/password.txt and rebuilding the docker image. Alternatively, an external identity provider such as Azure Active Directory or KeyCloak can be used. See: docs/azure_ad_setup.md for Azure AD setup instructions or docs/keycloak_setup.md for KeyCloak.
For complete instructions to install on a single Azure VM with Docker:
For complete instructions to install on AWS Kubernetes Service (EKS):
For complete instructions to install on AWS Lambda:
For complete instructions to install on a single Azure VM with Docker:
For complete instructions to install on Azure Kubernetes Service (AKS):
For complete instructions to install on a desktop or local server:
For complete instructions to install on DCOS:
Setting up docker:
Post install setup and testing:
Authorization, ACLs, and Role Based Access Control (RBAC):
As a REST service, clients be developed using almost any programming language. The test programs under: hsds/test/integ illustrate some of the methods for performing different operations using Python and HSDS REST API (using the requests package).
The related project: https://github.com/HDFGroup/h5pyd provides a (mostly) h5py-compatible interface to the server for Python clients.
For C/C++ clients, the HDF REST VOL is a HDF5 library plugin that enables the HDF5 API to read and write data using HSDS. See: https://github.com/HDFGroup/vol-rest. Note: requires v1.12.0 or greater version of the HDF5 library.
HSDS only modifies the storage location that it is configured to use, so to uninstall just remove source files, Docker images, and S3 bucket/Azure Container/directory files.
Create new issues at http://github.com/HDFGroup/hsds/issues for any problems you find.
For general questions/feedback, please use the HSDS forum: https://forum.hdfgroup.org/c/hsds.
HSDS is licensed under an APACHE 2.0 license. See LICENSE in this directory.
VM Offer for Azure Marketplace. HSDS for Azure Marketplace provides an easy way to setup a Azure instance with HSDS. See: https://azuremarketplace.microsoft.com/en-us/marketplace/apps/thehdfgroup1616725197741.hsdsazurevm?tab=Overview for more information.