architecture-on-a-page

[bwalsh]

Overview

The intent of this document is to provide a high level scope, define a requirements-stack, and illustrate an architectural vision for the effort. The core practices we followed were:

• Active Stakeholder Participation
• Collective Ownership
• Display Models Publicly
• Model In Small Increments
• Prove It With Code
• Use The Simplest Tools
• Create Simple Content
• more ...

'portal-proxy'

Functionality

Order	Description	Comment
1	Implement project security	Missing feature necessary for workflows and cohort discovery (currently mocked within the UI. target ohsu ldap for authorization and authentication)
2	Load all of OICR	Currently we've loaded only AML projects
3	release ohsu.dcc	demo: 1)branded ohsu.dcc portal, 2) ohsu.ldap & user roles 3) refreshed data[baml, icgc] pancreatic data is not in scope

from roadmap

Deployment

• The dcc-portal-ui, euler.api and openstack.keystone components are self contained in a docker container, quick start

Dataflows

• (0) - the projects' meta data and files are loaded into dcc (aka 'release')
• (1) - an admin user adds project structure using openstack command line
• (2) - a research user logs in and provides their ldap username and password.
• (3) - the euler.api calls keystone to get a token and returns it as a JWT (json web token)
• (4) - the user makes any /api call, the euler.api validates token and applies filters

Keystone data flows

Conceptual sequence diagram. All services [swift, compute, image, and euler] work the same way. Keystone is to openstack as the mitre service is to CCC.

Dependencies

• openstack versions
  • Keystone: VERSION=10.0.0 https://github.com/openstack/keystone/archive/${VERSION}.tar.gz
  • Openstack CLIENT_VERSION=3.2.0 pip install python-openstackclient==${CLIENT_VERSION}
• keystone system account
  • Ability to create domain & projects
• exastack
  • euler nodes needing certs and dns entries in CCC domain
  • dms_development
  • dms-staging
  • dms-prod
  • all communications via standard http, https ports
  • all hosts need access ingress OHSU-SECURE net and egress to ohsu.ldap
  • all hosts need dns (already done- check.)

'add-a-file'

Functionality

Order	Description	Comment
4	Implement lightweight add a file	Missing feature necessary for workflows (includes ccc_client. limited to repository functionality. release is out of scope)
5	Implement OHSU data store	While we have demonstrated loading and searching OHSU data, we have not demonstrated access to the BAML or other OHSU data (eg. downloading, workflow, BAM Stats,etc.) (aka object store)

Deployment

• The dcc-portal-ui, euler.api and openstack.keystone components are self contained in a docker container, quick start

Dataflows

• (0) - the projects' meta data and files are loaded into dcc (aka 'release')
• (1,2) - An authenticated user adds a file to the object store, and the euler plugin picks up meta data about the container, object(file) and user account and posts it to euler's api service for processing.
• (3) - The euler api server consumes the event and composes a Resource message and publishes it to a kafka topic. A kafka component picks up the message and uses existing calls to the dcc-server(which depends on es,mongo) to retrieve context data and creates a file centric document .
• (4) - An authenticated user, using the either the portal or a command line tool, queries for resources or metadata. The euler api component filters and redacts those calls.
• (5) - The euler api component relies on keystone for identity and authorization decisions.

Dependencies

• openstack versions
  • Swift: SWIFT_VERSION=2.10.0
  • Openstack CLIENT_VERSION=3.2.0
• swift system account
  • Ability to install euler swift listener
• clients have access to swift
• swift has access to euler.api

Current work

Deployment

• We move from the 'proof-of-concept' instances of keystone and swift to the ones already installed in exastack.
• The openstack.keystone components are removed from the docker container and injected into container via docker-compose-deploy.yml, and .env quick start
• The euler.listener is configured and deployed into exastack-00.ohsu.edu swift instance

Functionality

Order	Description	Comment
6	Common ETL with BMEG	Integrate event handling. Leverage same infrastructure (OHSU Kafka)
7	Load additional OHSU projects	Other projects (from CGD?) should be loaded to complement BAML

Dependencies

• openstack versions of swift and keystone and system accounts above available in exastack-00.ohsu.edu
• deprecate local instances of swift and keystone for environments other than development

Backlog

Functionality

The stakeholders and sponsors have identified inter-institutional collaboration as a goal. These features have not been prioritized and approved, but are captured here to provide roadmap.

Order	Description	Comment
8	Federate dcc api	Implement federation layer above dcc-server
9	Non OHSU sites	Support for separately administered keystone instances
10	Cloud datasets	Capture meta data about cloud datasets and workflows

un-groomed features

• keystone shibboleth integration
• keystone federation
• euler api federation
• s3, google bucket webhooks

[bwalsh]

@mayfielg @k1643 @kellrott Please review this draft document in the context of EUL-36

[grmayfie]

First architecture diagram: looks very good, except we're missing an explicit definition of the ports each piece is using to communicate with the rest of the setup. In EUL-36, we said we wanted to describe those definitively as well. It's helpful to document what ports need to be available externally on which floating IPs, since ITG needs to handle that for us. What are these diagrams drawn in? I'd be happy to make that addition, if I can edit the documents the diagrams came from.

First data-flow diagram: from everything I remember about the dcc-portal-server, it never queries mongo. It only queries elastic, and mongo is only ever used as a resource during the release pipeline. If I'm correct, then this diagram is a bit misleading. Am I missing some detail of the server that does use mongo directly? I'm also a bit unclear on what 'config' in the openstack, keystone box is referring to.

I'd life to clarify the 'future sprints' section a bit. The separation from one dotted box to two from the first diagram to the second refers to moving from using the proof-of-concept instances of keystone and swift to the ones already installed in exastack and ITG, correct?

This backlog appears to deal solely with federation (of authentication, cloud storage, dcc-server instances). It would be clearer to say that explicitly as the over-arching epic of the ungroomed backlog.

More specifically, Are keystone-with-shibboleth and keystone federation different options for addressing authentication of non-OHSU personnel? Or are they supposed to work in conjunction? Or is that unknown at this time and an issue to address when we get to that stage? Also, I believe "s3, google bucket webhooks," is referring to the issue of federation across multiple physical locations, and thus multiple cloud storage sites. At this time, is swift capable of federation across swift instances already, or no? It might be intelligent to include that as a story first, before we start to deal with inclusion of other softwares.

[bwalsh]

@mayfielg Thanks for comments. Please check to ensure that I've addressed them.

it never queries mongo

removed mongo from dataflows

I'd like to clarify the 'future sprints' section a bit.

added clarification

This backlog appears to deal solely with federation

added features section

[grmayfie]

I think the only remaining question is the bit regarding what's proposed for shibboleth and keystone federation.

Are keystone-with-shibboleth and keystone federation different options for addressing authentication of non-OHSU personnel or are they supposed to work in conjunction? Or is that unknown at this time and an issue to address when we get to that stage?

[bwalsh]

Agreed. It is an evolving architecture. In the next sprint planning we can discuss w/ Adam M + Kyle + Paul H.

[grmayfie]

+1