Data Platforms

This repository gathers together components that take part in the Commons Alliance. These components describe interfaces and features that can be assembled to create "data platforms" useful for storing, and performing reproducible analyses on data and metadata.

This is a living document.

Note, if you are viewing this on github, the images may be cached, please visit:

https://databiosphere.github.io/data-platforms/

For more background read the Data Biosphere post.

Visit the DataBiosphere github organization.

Metadata Serialization

Communication between data platforms requires that metadata are serialized in a useful and predictable manner. This document describes approaches and case studies in use by some components.

View the Metadata Serialization document.

Identifier Interoperability

When the same metadata are present in multiple locations, it is critical to provide guarantees of identity that are useful and portable. This document describes approaches to presenting and using interfaces that allow identifiers to be usefully exchanged.

View the Identifier Interoperability document.

Prototype

The various components coordinate to create a platform useful for data analysis.

Digital Object Catalog

Provides clients and services access to resources available in object stores. Digital objects can be files and the catalog itself maintains a registry of locations to find the files, as well as minimal metadata.

GUID Resolver

Allows globally unique identifiers to be "resolved" to digital objects. For more information please refer to Identifier Interoperability.

Namespace Service

Identifiers can be given different namespaces or "prefixes". The namespace service allows commons members to easily manage GUIDs across projects and domains. For more information please refer to Identifier Interoperability.

Data Access

Once data have been discovered they must be localized, which requires interacting with object stores and performing authentication, authorization, downloading, and transfer as necessary.

Access Control

To guarantee authority and authenticity of requests, some access control services are provided. These services will at least be able to identify a client and delegate authority to the access control system of choice.

Analytical Engine

Software which can orchestrate and execute computational tasks in heterogeneous computing environments.

Tool Repository

A resource which contains templates of reusable computational tasks that can be directed at new data, and then executed by the Analytical Engine.

Workspaces

Clients accessing a commons infrastructure should be able to manage data for secondary and tertiary data analysis.

Indexing and Search

Data in commons infrastructure should be findable using Search mechanisms. Indexing makes data available for search.

Ontology

A controlled vocabulary informs indexers and or querying applications to make metadata usable.

Metadata Indexer

Metadata made available by a platform is indexed into a store. Indexers allow data to be made findable using a structured document scheme.

Metadata Querying

Once metadata have been indexed into a platform, these indices are made available by services that allow queries to be formed against the metadata.

Portal

Commons infrastructure should provide interfaces to make data easily findable. Once data has been found in a portal, it can be added to a workspace.

Application

Applications combine a variety of Commons components to carry out specific tasks.

Commons Alliance Components

Source Code Repository Table

Links to source code repositories for implementations are provided below:

Applications marked with a * implement a standard interface being developed with the GA4GH. Clients can interact with these applications using an open protocol

• indexd, dos-azul-lambda and Cromwell implement the Data Object Service.
• Dockstore and Agora implement the Tool Registry Service.
• Cromwell implememnts the Workflow Execution Service

UChicago CDIS

The University of Chicago, CDIS groups presents software for easily managing the submission and access control of bioinformatics and medical informatics data in cloud environments.

UC Santa Cruz Computational Genomics Platform

Broad Institute

This section is in progress

Development

This document is under active development. If you feel misrepresented or something has been miscommunicated, please open an issue or make a Pull Request!

Editing diagrams

The program used to edit the "dia" files is dia.

Github caches images when they display READMEs so be sure to check the actual file if it seems out of date!