This repository gathers together components that take part in the Commons Alliance. These components describe interfaces and features that can be assembled to create "data platforms" useful for storing, and performing reproducible analyses on data and metadata.
This is a living document.
Note, if you are viewing this on github, the images may be cached, please visit:
https://databiosphere.github.io/data-platforms/
For more background read the Data Biosphere post.
Visit the DataBiosphere github organization.
Communication between data platforms requires that metadata are serialized in a useful and predictable manner. This document describes approaches and case studies in use by some components.
View the Metadata Serialization document.
When the same metadata are present in multiple locations, it is critical to provide guarantees of identity that are useful and portable. This document describes approaches to presenting and using interfaces that allow identifiers to be usefully exchanged.
View the Identifier Interoperability document.
The various components coordinate to create a platform useful for data analysis.
Provides clients and services access to resources available in object stores. Digital objects can be files and the catalog itself maintains a registry of locations to find the files, as well as minimal metadata.
Allows globally unique identifiers to be "resolved" to digital objects. For more information please refer to Identifier Interoperability.
Identifiers can be given different namespaces or "prefixes". The namespace service allows commons members to easily manage GUIDs across projects and domains. For more information please refer to Identifier Interoperability.
Once data have been discovered they must be localized, which requires interacting with object stores and performing authentication, authorization, downloading, and transfer as necessary.
To guarantee authority and authenticity of requests, some access control services are provided. These services will at least be able to identify a client and delegate authority to the access control system of choice.
Software which can orchestrate and execute computational tasks in heterogeneous computing environments.
A resource which contains templates of reusable computational tasks that can be directed at new data, and then executed by the Analytical Engine.
Clients accessing a commons infrastructure should be able to manage data for secondary and tertiary data analysis.
Data in commons infrastructure should be findable using Search mechanisms. Indexing makes data available for search.
A controlled vocabulary informs indexers and or querying applications to make metadata usable.
Metadata made available by a platform is indexed into a store. Indexers allow data to be made findable using a structured document scheme.
Once metadata have been indexed into a platform, these indices are made available by services that allow queries to be formed against the metadata.
Commons infrastructure should provide interfaces to make data easily findable. Once data has been found in a portal, it can be added to a workspace.
Applications combine a variety of Commons components to carry out specific tasks.
Links to source code repositories for implementations are provided below:
Component | Broad | UChicago CDIS | UCSC CGP |
---|---|---|---|
Digital Object Catalog | |||
GUID Resolver | indexd* | dos-azul-lambda* | |
Namespace Service | indexd* | ||
Data Access | fence | cgp-data-store | |
Access Control | |||
Authorization | sam bond | fence | |
Authentication | sam bond | fence | |
Analytical Engine | Cromwell** Leonardo |
toil | |
Tool Repository | Agora* | Dockstore* | |
Workspaces | Rawls | jupyterhub | |
Indexing and Search | Orchestration | ||
Ontology | datadictionary | ||
Metadata Indexer | Orchestration | sheepdog | azul-indexer |
Metadata Querying | Orchestration | peregrine | azul-webservice |
Portal | Firecloud | windmill | boardwalk |
Application | xena |
Applications marked with a *
implement a standard interface being developed with the GA4GH.
Clients can interact with these applications using an open protocol
The University of Chicago, CDIS groups presents software for easily managing the submission and access control of bioinformatics and medical informatics data in cloud environments.
This document is under active development. If you feel misrepresented or something has been miscommunicated, please open an issue or make a Pull Request!
The program used to edit the "dia" files is dia.
Github caches images when they display READMEs so be sure to check the actual file if it seems out of date!