Closed kirillsc closed 1 month ago
@kirillsc - thanks for your contribution proposal! I'm going to guide you and team through the contribution process.
As next step, we'll organize a meeting with you and the team, to give you a high level walkthrough of the process and refine the checklist mentioned above.
I'll also engage with our Technical Steering Committee to ask for their approval. Thank you!
@kirillsc - thanks for the contribution proposal. I see merit in this approach and what is not clear yet is the maturity / fit of this solution. Do you know any FI that is actually using it in production environment? Knowing this help steer the roadmap.
Thanks you!
Approved, but please make sure the roadmap is documented. I know there are plans for the project and to extend support to the other providers but this needs to be incorporated into the roadmap
Great news! I look forward to working together on this project!
Announcement sent, see https://groups.google.com/a/finos.org/g/announce/c/b7tAlU5rmBI.
The contribution process is completed (there are few leftovers that are in progress but they are not blockers).
Congratulations to the Amazon team, and thanks for contributing HTC Grid to FINOS!
Should this issue be closed @TheJuanAndOnly99 or anything left to do?
@mindthegab Yes. Closing issue. There are still a few tasks that need to be done on our end that I'll keep track of internally.
Please note that only FINOS members can propose new projects. If you're interested in membership, see https://www.finos.org/membership-benefits#become-a-member.
Onboarding Process
Completing an onboarding of a project into FINOS requires following these 5 main steps:
1. Describing The Contribution
This is a list of questions that need to be answered by the contributor in order to allow a new project to pass to the approval stage of onboarding.
Business Problem
Financial Services Institutions (FSIs) perform a large volume of computations as part of their daily operations, such as pricing their products, monitoring, and controlling their risk exposure. These computations are typically executed on compute grids, which are paramount components of the FSIs' infrastructure today. Historically, compute grids were running on on-premises infrastructure and these systems can be very large, complex and expensive to operate.
Market volatility and new regulatory requirements, such as FRTB and Solvency II, are driving a significant increase in compute load today. Aging on-premises infrastructure is both expensive to retain and slow to scale up. To provide better service to their customers, financial service institutions increasingly desire faster intra-day computational results, on-demand analytics, and innovation. All of these factors prompt FSIs to re-invent their computing grids and explore modernisation pathways in the cloud.
Furthermore, FSI workloads often involve a high number of short-running tasks. It is not uncommon to process more than 100 million tasks per day, with a large portion of these tasks taking less than 2 seconds to compute. These unique characteristics make many existing grid schedulers unsuitable for these workloads.
Proposed Solution
The High Throughput Compute Grid project (HTC-Grid) is a container-based cloud native HPC/Grid solution. The project provides a reference architecture that can be used to build and adapt a modern high throughput compute solution using underlying AWS services, allowing users to submit high volumes of short and long running tasks.
HTC-Grid addresses the massive computational demands by combining very high throughput scheduling (over 30,000+ submissions a second), low latency (under 300 ms) and seamless infrastructure orchestration. HTC-Grid provides uniform architecture suitable to handle large scale nightly batch workloads and intra day workloads that require near real-time response times.
By using a modular architecture made up of managed services such as Amazon EKS, Amazon DynamoDB, Amazon SQS, (or similar services from other CSPs) and the use of Spot instances, HTC-Grid is able to dynamically scale computing resources and meet the demanding requirements of FSIs.
Tentative Roadmap
Initial plan is to move existing HTC-Grid repository from AWS-Labs to FINOS.
Current State
HTC-Grid source code - https://github.com/awslabs/aws-htc-grid HTC-Grid workshop & documentation - https://catalog.us-east-1.prod.workshops.aws/workshops/6b17d1f3-419d-4c05-821f-1dd2c3488d6f
Existing Materials
If materials already exist, provide a link to them that Foundation staff can access - if it's in a private GitHub.com repositories, you should invite the finos-admin user with R/O permissions to those repositories
Development Team
Maintainers
Who will be the project maintainer(s)? Provide full name, affiliation, work email address, and GitHub / GitLab username.
Confirmed contributors
If applicable, list all of the individuals that have expressed interest in and/or are committed to contributing to this project, including full name, affiliation, work email address, and GitHub.com username
Target Contributors
Describe the contributor profile (background, position, organization) you would like to get contributions from.
Project Communication Channel(s)
Understanding FINOS Onboarding Requirements
As a project onboarding into FINOS, you will need to familiarize yourself and your contributor team with the following materials:
Record The Contribution (FINOS Infra)
2. Approval
The FINOS Technical Oversight Committee (TOC) is responsible for approving FINOS project contributions; feel free to check their contribution principles.
If needed, the TOC will request a follow up either via GitHub Issue comments or by inviting project leads to one of their recurrent meetings.
Tasks (for FINOS Infra/TOC)
ready-for-tsc
labelTOC Findings / Report
TOC to enter findings summary here.
3. Preparing For Onboarding
Before the FINOS team can onboard your project, there are a few housekeeping that need to be taken care of. These must be completed by the contributor, with help if required from the FINOS Infra.
Kick-off meeting
Logo / Trademarks
help@finos.org
(if needed)FINOS Project Blueprint
CONTRIBUTING.md
LICENSE
(replace{}
placeholders)Add documentation here
4. FINOS Onboarding
This is performed by FINOS Infra once the three previous stages are complete, with support from the contributor and the FINOS Infra team.
Maintainers, Contributors and CLAs
<project-name>-maintainers
GitHub team and invite usersValidation (only if code is contributed)
Admin
to all repositories to transfer@kirillsc - All dependencies in requirements.txt are unpinned; is there any reason for that? If not, we'd recommend to follow Python best practices and pin dependencies.
bandit
for python,tflint
for Terraform/OpenTofu files andcfn-lint
for Cloudformation files)No Category X licenses found
@kirillsc - it would be great to create a GitHub Action to run this continuously, every time the requirements.txt file gets updated. Is this something you and team could work on?
Code transfer
main
(instead ofmaster
)finos-admins
(Maintain
role) andfinos-staff
(Triage
role) team permissionsProject Communication Channel(s)
Repository setup
staging
branch onfinos/finos-landscape
finos/metadata
changes on master (will udpdatelandscape.yml
infinos/finos-landscape
)staging
branch onfinos/finos-landscape
finos
Require a pull request before merging
)5. Announcement
(Lead: Project Lead and FINOS Infra team)