[Project] Rebuild IDMS Stack

[mikethicke]

Description

Rebuild a containerized version of the Knowledge Commons IDMS stack that is suitable for local development and deployment to production

Background

The current IDMS stack is composed of several services distributed across 3 EC2 instances (Miro diagram).

Main Services

• Proxy
  • SATOSA
• Registry
  • COmanage
  • Shibboleth (It might simplify things to replace this with SimpleSAMLphp?)
  • OpenLDAP
• IDPOLR
  • SimpleSAMLphp
• AWS RDS
  • Registry Database

These services have various configurations and plugins / modules / microservices specific to our setup. These customizations are not consistently in version control or synchronized between development and production. Many are out-of-date. We would like to change that.

There are currently two copies of this stack---dev and prod. We would like to move to a model with local development, and cloud-based staging and production. We might consider moving production and / or staging to a serverless solution, if that is more cost-effective than EC2.

Deliverables

• All Commons IDMS code should exist in a single repository (this one).
• The configurations, custom code, and plugins for services should be organized by directory. Eg:

├-/
  ├-metadata/
  | ├-dev-metadata.xml
  | ├-prod-metadata.xml
  |
  ├-SATOSA/
  | ├-hcdiscovery.yaml
  | ├-idp_metadata_attribute_store.yaml
  | ├-src/
  |   ├-hcdiscovery.py
  |   | ...
  |
  ├-COmanage/
  |  ├-Config/
  |  ├-Plugin/
  |  ├-webroot/
  ...

• Each necessary service should exist in a container.
• Where possible, containers should be based on standard production or development containers (eg. on Docker Hub).
• The build process (eg. docker compose up or lando start) should build containers for each service and mount the Commons code and configurations appropriately. The containers should network together and operate in (local) dev, staging, and production environments.
• Environment variables should control the build process to target each environment.
• There should be a way to test the stack locally without WordPress (eg. static HTML pages with links for registering & logging-in test users, and reporting successful logins).
• The stack should work with the containerized Commons WordPress stack (which will be developed in parallel).

Target Timeline

• Project Start: 23-04-24
• Running Locally w/ RDS: 23-06-01
• Full stack locally: 23-06-15
• Deploy to AWS (development): 23-07-15
• Ready for QA: 23-08-01
• Deploy to production: 23-10-01

Responsibilities

Development Leads: Dimitris & Brian

[mikethicke]

Big Stages

• [ ] Get IDMS stack working locally with development configuration and AWS RDS database
• [ ] Local IDMS stack with local testing database
• [ ] Works with multiple ( development / production ) configurations
• [ ] Deploys to staging / production on AWS

Getting started

• [ ] Figure out all of the essential services
  • Look at Miro boards, developer documentation. Verify that it is correct and complete.
• [ ] Gather configuration files and other local customizations
• [ ] Figure out how to use updated versions of the services (eg. SATOSA is currently way of out date)
• [ ] ????
• [ ] Get the local stack running

[mikethicke]

Resources:

• HC IDMS Stack: https://miro.com/app/board/uXjVPH6GGBA=/
• Identity Flow: https://miro.com/app/board/uXjVPNBgBWk=/
• IDMS stack documentation: https://github.com/MESH-Research/hc-admin-docs-support/blob/main/documentation/developer/documentation/idms-stack.md
• Humanities Commons Handbook: https://docs.google.com/document/d/1C13xaEnUPJ_dLUGQDbXv5Cr-E738pgg9dgU3enJyeEQ/edit#heading=h.uutzyhyfuu7s
• COmanage documentation: https://spaces.at.internet2.edu/display/COmanage/Home