Unity SDS Processing Service (U-SPS)
The Unity SDS Processing Service facilitates large-scale data processing for scientific workflows.
This repository contains high level information (such as documentation, change log, etc.) about the U-SPS software stack. The actual U-SPS code is contained within the following set of GitHub repositories:
- U-SPS Prototype: Terraform scripts to deploy the U-SPS cluster (either the HySDS or Airflow implementations)
- U-SPS Workflows: Examples of CWL workflows that can be executed on a U-SPS cluster
- U-SPS API: The API used to manage a U-SPS cluster
- U-SPS Register Job: Implementation of the WPS-T API with respect to the supported U-SPS back-ends
Features
- Deployment and execution of scientific data processing algorithms via OGC WPS-T API.
- CWL standard for workflow encoding.
- API management for cluster resources.
- Docker-packaged applications interacting within a Kubernetes cluster.
- U-SPS supports HySDS and Apache Airflow implementations.
Contents
Quick Start
This guide provides a quick way to get started with our project. Please see our docs for a more comprehensive overview.
Requirements
- Docker 20.10 or higher
- Kubernetes 1.20 or higher
- Terraform 0.14 or higher
Setup Instructions
- Follow the U-SPS Setup Guide.
Run Instructions
- Initialize the Terraform scripts.
- Apply the Terraform configuration.
- Monitor the U-SPS status on Kubernetes.
- Check data processing results.
Usage Examples
- Tutorial: Executing the L1B workflow
- Tutorial: Using Job Labels
- Tutorial: Testing the SPS Prewarm API
Build Instructions (if applicable)
N/A
Test Instructions (if applicable)
N/A
Changelog
See our CHANGELOG.md for a history of our changes.
Visit our releases page for versioned releases.
Frequently Asked Questions (FAQ)
Questions about our project? Please see our: FAQ
Contributing
Interested in contributing to our project? Please see our: CONTRIBUTING.md
License
See our: LICENSE
Support
Key points of contact are: