High Throughput Compute Grid (HTC-Grid) Software Project Contribution and Onboarding

[kirillsc]

Please note that only FINOS members can propose new projects. If you're interested in membership, see https://www.finos.org/membership-benefits#become-a-member.

Onboarding Process

Completing an onboarding of a project into FINOS requires following these 5 main steps:

1. Describing the Contribution led by contributor
2. Approval led by FINOS TOC
3. Preparing for Onboarding led by contributor
4. Onboarding completed by FINOS Infra
5. Announcement led by FINOS Marketing

1. Describing The Contribution

This is a list of questions that need to be answered by the contributor in order to allow a new project to pass to the approval stage of onboarding.

Business Problem

Financial Services Institutions (FSIs) perform a large volume of computations as part of their daily operations, such as pricing their products, monitoring, and controlling their risk exposure. These computations are typically executed on compute grids, which are paramount components of the FSIs' infrastructure today. Historically, compute grids were running on on-premises infrastructure and these systems can be very large, complex and expensive to operate.

Market volatility and new regulatory requirements, such as FRTB and Solvency II, are driving a significant increase in compute load today. Aging on-premises infrastructure is both expensive to retain and slow to scale up. To provide better service to their customers, financial service institutions increasingly desire faster intra-day computational results, on-demand analytics, and innovation. All of these factors prompt FSIs to re-invent their computing grids and explore modernisation pathways in the cloud.

Furthermore, FSI workloads often involve a high number of short-running tasks. It is not uncommon to process more than 100 million tasks per day, with a large portion of these tasks taking less than 2 seconds to compute. These unique characteristics make many existing grid schedulers unsuitable for these workloads.

Proposed Solution

The High Throughput Compute Grid project (HTC-Grid) is a container-based cloud native HPC/Grid solution. The project provides a reference architecture that can be used to build and adapt a modern high throughput compute solution using underlying AWS services, allowing users to submit high volumes of short and long running tasks.

HTC-Grid addresses the massive computational demands by combining very high throughput scheduling (over 30,000+ submissions a second), low latency (under 300 ms) and seamless infrastructure orchestration. HTC-Grid provides uniform architecture suitable to handle large scale nightly batch workloads and intra day workloads that require near real-time response times.

By using a modular architecture made up of managed services such as Amazon EKS, Amazon DynamoDB, Amazon SQS, (or similar services from other CSPs) and the use of Spot instances, HTC-Grid is able to dynamically scale computing resources and meet the demanding requirements of FSIs.

Tentative Roadmap

Initial plan is to move existing HTC-Grid repository from AWS-Labs to FINOS.

Current State

HTC-Grid source code - https://github.com/awslabs/aws-htc-grid HTC-Grid workshop & documentation - https://catalog.us-east-1.prod.workshops.aws/workshops/6b17d1f3-419d-4c05-821f-1dd2c3488d6f

Existing Materials

If materials already exist, provide a link to them that Foundation staff can access - if it's in a private GitHub.com repositories, you should invite the finos-admin user with R/O permissions to those repositories

• [x] GitHub / GitLab Repository (yes)
  • [x] URL for the repository https://github.com/awslabs/aws-htc-grid
  • [x] Project Name (HTC-Grid)
  • [x] @finos-admin has been given read-only permissions if private [(public)]
• [x] Is Continuous Integration used? [(GitHub Actions and AWS CodePipeline)]
• [x] Was the project ever released? [(yes)]
  • [x] If so, are releases public? [(yes)]
  • [x] And what's the latest released version? [(v0.4.3)]
• [x] Existing Project Documentation [( https://catalog.us-east-1.prod.workshops.aws/workshops/6b17d1f3-419d-4c05-821f-1dd2c3488d6f)]
• [ ] Does the name have a registered trademark? (yes / no)
• [x] Is there a logo? [no]
• [x] High-Level Presentation prepared for Technical Oversight Committee (~15 mins) (https://www.youtube.com/watch?v=dJ0GTgz5Nrs)
• [x] Are meetings currently held for the project? [no]
• [x] Are meeting minutes, agenda and attendance tracked? [no]

Development Team

Maintainers

Who will be the project maintainer(s)? Provide full name, affiliation, work email address, and GitHub / GitLab username.

Name	Affiliation	Work Email Address	Github / GitLab username
Clement Rey	Amazon	clemerey@amazon.fr	clementrey-dev
Flamur Gogolli	Amazon	flamurg@amazon.co.uk	fgogolli
Kirill Bogdanov	Amazon	kirillb@amazon.ch	kirillsc

Confirmed contributors

If applicable, list all of the individuals that have expressed interest in and/or are committed to contributing to this project, including full name, affiliation, work email address, and GitHub.com username

Name	Affiliation	Work Email Address	Github / GitLab username
Clement Rey	Amazon	clemerey@amazon.fr	clementrey-dev
Flamur Gogolli	Amazon	flamurg@amazon.co.uk	fgogolli
Kirill Bogdanov	Amazon	kirillb@amazon.ch	kirillsc

Target Contributors

Describe the contributor profile (background, position, organization) you would like to get contributions from.

Project Communication Channel(s)

• [ ] Contributor to ask maintainers which communications channels they'd like to use:
• Asynchronous
  • [x] GitHub Issues [https://github.com/awslabs/aws-htc-grid/issues]
  • [ ] GitHub Discussions (public)
  • [ ] GitHub Team Discussions (consisting of the above described contributors)
  • [ ] Public
  • [ ] Private
  • [ ] Mailing-list (groups.io)
  • [ ] FINOS Slack Channel (consisting of the above described contributors)
  • [ ] General (public) (supply channel name)
  • [ ] Leadership (private) (supply channel name)
• Synchronous
  • [ ] Recurring meetings

Understanding FINOS Onboarding Requirements

As a project onboarding into FINOS, you will need to familiarize yourself and your contributor team with the following materials:

• FINOS overview (if necessary)
• FINOS Maintainers cheatsheet
• FINOS Project/Standards Governance
• FINOS Project Lifecycle

Record The Contribution (FINOS Infra)

• [ ] Add project to LFX PCC

2. Approval

The FINOS Technical Oversight Committee (TOC) is responsible for approving FINOS project contributions; feel free to check their contribution principles.

If needed, the TOC will request a follow up either via GitHub Issue comments or by inviting project leads to one of their recurrent meetings.

Tasks (for FINOS Infra/TOC)

• [x] Prioritise this issue on the TOC Backlog
• [x] Assign this issue to @colineberhardt
• [x] Add the ready-for-tsc label
• [ ] TOC to invite contributors to present their project
• [ ] FINOS TOC approves/rejects the contribution
• [ ] Update project in LFX with Status="Engaged"

TOC Findings / Report

TOC to enter findings summary here.

3. Preparing For Onboarding

Before the FINOS team can onboard your project, there are a few housekeeping that need to be taken care of. These must be completed by the contributor, with help if required from the FINOS Infra.

Kick-off meeting

• [x] Set up kick-off meeting with project leads
• [x] Run kick-off meeting
  • [x] Walk through the checklist in part 1, ensure all the questions are answered and remove items that don't apply
  • [ ] Plan presentation to the TOC

Logo / Trademarks

• [ ] Sign the project contribution agreement to allow FINOS to act on behalf of the contributor for accounts related to the project (e.g., GitHub, domain names, social media) and to optionally manage trademark assignment
• [ ] Request logo design from help@finos.org (if needed)
• [ ] The codebase doesn’t include any patent or copyright that conflicts with FINOS Governance and bylaws. (Infra team to validate with FINOS Legal team if anything important is raised)

FINOS Project Blueprint

• [ ] finos-admin is Admin of the GitHub repository to transfer
• [ ] Apply project blueprint contents - see our docs on collaboration infrastructure
  • [ ] README.md file contains the sections in the README Template
  • [ ] Project FINOS badge in README (incubating)
  • [ ] OpenSSF Passing Badge
  • [ ] License in README
  • [ ] Contributing in README
  • [ ] CONTRIBUTING.md
  • [ ] LICENSE (replace {} placeholders)
  • [ ] Ensure that the proper project governance is in the CONTRIBUTING.md file
• [ ] All incubating criteria are checked and documented below

Add documentation here

4. FINOS Onboarding

This is performed by FINOS Infra once the three previous stages are complete, with support from the contributor and the FINOS Infra team.

Maintainers, Contributors and CLAs

• [ ] Identify other existing contributors (assuming there's a contribution history (eg Git history)
• [ ] Check if maintainers and other contributors are all covered by FINOS CLA
• [ ] Engage with FINOS Legal team to figure out what’s needed to cover all maintainers and contributors with FINOS CLA
• [ ] Reach out to contributors and employers to coordinate CLA signatures
• [ ] Invite GitHub usernames to GitHub FINOS Org
• [ ] Create <project-name>-maintainers GitHub team and invite users

Validation (only if code is contributed)

• [ ] Backup (even with screenshot) GitHub permissions of the repository to transfer
• [ ] Check GitHub repository transfer requirements:
  • [ ] finos-admin has Admin to all repositories to transfer
  • [ ] finos-admin ia allowed to transfer repositories out of the org
  • [ ] if the repository is owned by a user (and not an org), the user must be able to transfer the repository to finos-admin
• [ ] The codebase doesn’t have HIGH or CRITICAL CVEs across direct and transitive libraries

  python3 -m venv .
  ./bin/pip install safety
  ./bin/safety check --full-report -r requirements.txt

@kirillsc - All dependencies in requirements.txt are unpinned; is there any reason for that? If not, we'd recommend to follow Python best practices and pin dependencies.

• [x] The codebase is scanned against security issues (using bandit for python, tflint for Terraform/OpenTofu files and cfn-lint for Cloudformation files)
• [x] The codebase doesn’t have any unfriendly licenses across direct and transitive libraries

python3 -m venv .
./bin/pip install pip-licenses
./bin/pip-licenses --allow-only="ISC License (ISCL); BSD License; Apache Software License; MIT License; Python Software Foundation License; Mozilla Public License 2.0 (MPL 2.0)"

No Category X licenses found

@kirillsc - it would be great to create a GitHub Action to run this continuously, every time the requirements.txt file gets updated. Is this something you and team could work on?

• [ ] (optional - if a build system is provided) The build process runs successfully
• [ ] The codebase doesn’t include any patent or copyright that conflicts with FINOS Governance and bylaws (to be validated with FINOS Legal team)
• [ ] All incubating criteria are met (review documentation provided above)

Code transfer

• [ ] Transfer all code assets as GitHub repositories under github.com/finos
• [ ] Rename main branch to main (instead of master)
• [ ] Configure finos-admins (Maintain role) and finos-staff (Triage role) team permissions

Project Communication Channel(s)

• [ ] Create the identified communication channels during infra set up
• [ ] Create mailing-list on lists.finos.org (if requested)
  • [ ] Enable Hubspot Sync for all project mailing lists created
  • [ ] Aggregate mailing lists to community@finos.org
  • [ ] Update marketing lists (see Email List and Email filter fields), particularly Hubspot all community list
• [ ] Create Slack channels if requested.
• [ ] Link communication channels linked front-and-center in the project README.md

Repository setup

• [ ] Enable EasyCLA
• [ ] Add project to metadata
• [ ] Add identities, orgs and affiliations to metadata (deprecated by EasyCLA)
• [ ] Add logo to FINOS landscape
  • [ ] Create staging branch on finos/finos-landscape
  • [ ] Merge finos/metadata changes on master (will udpdate landscape.yml in finos/finos-landscape)
  • [ ] Create PR from staging branch on finos/finos-landscape
  • [ ] Review Netlify preview
  • [ ] Merge PR
• [ ] Add project maintainers emails to finos-project-maintainers@finos.org list
• [ ] Add project maintainers GitHub usernames to the project-maintainers Team
• [ ] Ask @jgavronsky to mark contribution as "Active" within LF systems
• [ ] (best effort) Update release coordinates and code namespace to include finos
• [ ] Check that CVE (and preferably static code analysis, if applicable) scanning is in place
• [ ] Enable automated dependency update, using Renovate
• [ ] Enable branch protection (Require a pull request before merging)
• [ ] (optional) Enable meeting attendance tracking
• [ ] (optional) Onboard into legend.finos.org/studio

5. Announcement

(Lead: Project Lead and FINOS Infra team)

• [ ] Update the contribution status to "Active" by sending another email to LF Legal Representative with the name of the project and its new status.
• [ ] Lead maintainer works with FINOS marketing to send out announcement to announce@finos.org , checkout announcement template at the Contribution page
• [ ] Notify FINOS marketing (@grizzwolf + finos-marketing internal Slack channel)

[maoo]

@kirillsc - thanks for your contribution proposal! I'm going to guide you and team through the contribution process.

As next step, we'll organize a meeting with you and the team, to give you a high level walkthrough of the process and refine the checklist mentioned above.

I'll also engage with our Technical Steering Committee to ask for their approval. Thank you!

[brunodom]

@kirillsc - thanks for the contribution proposal. I see merit in this approach and what is not clear yet is the maturity / fit of this solution. Do you know any FI that is actually using it in production environment? Knowing this help steer the roadmap.

Thanks you!