Migration of CDM Release Process from codefresh to github actions

[chrisisla]

As agreed in the meeting of the CDM Technology Architecture Working Group on November 9th (#2484) the next initiative to be undertaken by the group is to migrate the existing release process from codefresh to github actions. This should make the release process more transparent to the community, and allow a broader group of the CDM maintainers to execute the process.

This Issue describes the scope that we will be using for the project.

[!IMPORTANT] This project will only be looking at the build/release technical process. The governance, approval, roles and responsibilities, content and scheduling of releases will not be considered as part of this work. The CDM Steering Working Group will define/agree any changes to these aspects of the release process.

In Scope

• Replicate existing codefresh based release process in github actions
  • Like for like functionality only
  • No new functionality will be added
  • Best endeavours will be made to not reduce functionality (see Note below)
• Define a transition plan
  • How to transfer the new process to the CDM Repository
  • When to do so i.e. dependencies for transition
  • How to execute the new process
  • Who will need to be made aware of the new process and how it will be presented
• Provide documentation that describes:
  • Each action, what it does and why it is necessary
  • Troubleshooting guide for known issues and resolutions for them where possible
  • How to execute the new process

[!NOTE] Minesh has confirmed that the tasks currently performed by codefresh are all available in github actions. However, we should cater for the fact that during the development we may encounter restrictions on what can be migrated to github actions. We will do our best not to reduce the functionality of the release process but will document any changes that we need to make.

Out of Scope

These are all out of scope for this project. These can be considered and addressed in future phases of work.

• New functionality
  • No additional functionality will be added at this stage
  • This includes, but is not limited to
  • Automation of releases
  • Enhanced permissions (see Roles and Responsibilities below)
• Roles and Responsibilities
  • Who can run the new process
  • How do we set up permissions to control roles
  • To be defined/agreed by the SWG
• Scheduling of releases
  • When to run the release process
  • Approval process for PRs to be included in releases
  • To be defined/agreed by the SWG

If you have any comments or suggested changes to the scope then please add a comment to this Issue and they will be considered for inclusion.

[chrisisla]

FYI: @dschwartznyc , @manel-martos , @osttra-j-joyce, @eacunaISDA , @brianlynn2 , @nicholas-moger

[minesh-s-patel]

Slides for meeting - CDM - Migration of the Release Process - Phase 1 29/01/2024 - 13:00 GMT

CDM Build and Release Process.pptx

[dschwartznyc]

Agenda for kick-off meeting: Jan 29, 8-10 am EST, 1-3 pm GMT, 2-4 pm CEST, 6:30-8:30 pm IST

1. Agenda, introductions, process: Dan, 5 minutes
2. In scope / out of scope: Chris, 10 minutes (see https://github.com/finos/common-domain-model/issues/2623)
3. Review the key project activities: Dan, 15 minutes

• Understand the current workflow
• Map and implement the current process in GitHub actions
• Document the on-process and off-process flows
• Run in parallel and confirm that the artifacts match
• Training
• Assign roles and responsibilities (ROTA)
• Cutover
  4. Review the current process, Minesh, 85 minutes (see https://github.com/finos/common-domain-model/files/14084405/CDM.Build.and.Release.Process.pptx)
  5. Next steps and wrap up, Dan, 5 minutes

[dschwartznyc]

Recap from the kick-off meeting Jan 29, 8-10 am EST, 1-3 pm GMT, 2-4 pm CEST, 6:30-8:30 pm IST

Comments and corrections are welcome.

In attendance (by name and GitHub Id or company): Arnau Oller (TradeHeader), Bramaramba Bhakthavatsalam (JPM), Brian Lynn (@brianlynn2), Chris Rayner (@chrisisla), Dan Schwartz (@dschwartznyc), Dhruva Sen Gupta (JPM), Eleonora Acuna (@eacunaISDA), Geogy Varughese (JPMC), Gerard Opazo (@gopazoTH), Harsha Valaturu (JPMC),James Joyce (@osttra-j-joyce), Manuel Martos (@manel-martos), Minesh Patel (@minesh-s-patel), Nicholas Moger (JPM), Pranabashish Deb (JPM),Tom Healey (@tomhealey-icma), Vishal P Naik (JPMC)

Agreed:

• The meeting was conducted under the FINOS Project Guidelines (https://community.finos.org/docs/governance/meeting-procedures/).
• The meeting was recorded to facilitate documenting the existing process.
• The scope of the project was narrowed to the migration of the Build Process.
• The migration will require committed action on the part of meeting participants.
• The project will be executed in stages with the first phase focused on migrating and documenting the following activities described in the CDM Release and Build process overview slides. Later stages may be executed in parallel.

• main_clone
• SetProperties
• ReleaseProperties
• SnapshotProperties

• The next meeting will be scheduled for Feb 5 at 9 am EST, 2 pm GMT, 3 pm CEST, 7:30 pm IST. The agenda for that meeting is to begin implementing the stage phase of the project. Committed participants: Arnau, Brian, Chris, Dan, Eleonora (TBC), Gerard, James, Manuel, Minesh, Tom
• Project work will be done in a repo separate from the main CDM project. There will not be an approval process for changes made to this repo. Once the project is completed, a PR will be made to the main repo.

Actions:

• Schedule the next meeting, Dan
• Create a repo to host the migration: Minesh. Done https://github.com/minesh-s-patel/common-domain-model-github-action-migration
• Provide Minesh with the GitHub Id's of those who will be actively participating in the first phase of the migration: Dan
• Confirm that this Task Force is operating under FINOS guidelines: Chris and Dan

Notes: current experience is that it takes approximately 2 hours for the build process to run when there are no issues. The timing for resolution of issues depends on the nature of the problem.

[manel-martos]

The proposed workflow file has been pushed into the hosted repository. Find it here.

[dschwartznyc]

Recap and next steps from meeting # 2 - Mar 26, 9 AM EDT, 1 PM GMT, 2 PM CEST

Comments and corrections are welcome.

In attendance (by name and GitHub Id or company): Arnau Oller (TradeHeader), Brian Lynn (@brianlynn2), Chris Rayner (@chrisisla), Dan Schwartz (@dschwartznyc), Eleonora Acuna (@eacunaISDA), Gerard Opazo (@gopazoTH), Hugo Hills (@hugohills), Manuel Martos (@manel-martos), Tom Healey (@tomhealey-icma)

Recap:

• Reiterated objectives
• Reviewed the (excellent) TradeHeader prototype for the first four actions
• Agreed the goal for the next checkpoint is revisions based on the below. Once completed, the first build item to be converted will be for the Java version. Note the intent to delegate implementation across the TAWG maintainers

Next steps:

• Confirm that the proposed approach for storing and accessing secrets meets FINOS guidelines (n.b., plan to use the same method as is used in the DSL repo): Dan
• Clean up the prototype and add to the repo: Manuel (done)
• Split the process into separate actions for building release candidates and for building releases. The former may be triggered automatically but production releases will be triggered manually: TradeHeader team
• Schedule the next meeting: Dan

[dschwartznyc]

Recap and next steps from meeting # 3 - Apr 10, 10 AM EDT, 3 PM BST, 4 PM CEST

Comments and corrections are welcome.

In attendance (by name and GitHub Id or company): Arnau Oller (TradeHeader), Brian Lynn (@brianlynn2), Chris Rayner (@chrisisla), Dan Schwartz (@dschwartznyc), Eleonora Acuna (@eacunaISDA), Gerard Opazo (@gopazoTH), Manuel Martos (@manel-martos), Minesh Patel (@minesh-s-patel), Tom Healey (@tomhealey-icma)

Recap:

• Secret data -- Confirmed plan to store the process user name in secrets -- Confirmed plan to try to use GitHub Actions native mechanism to address GPG requirements
• Prototype demo by TradeHeader -- includes the Build phase in addition to main_clone, SetProperties, ReleaseProperties, SnapshotProperties -- agreed to use GitHub actions automation for deployment (a small change to the CodeFresh approach)
• Ownership of implementation of remaining elements -- First phase --- @chrisisla: BuildDaml, DeployDaml --- @dschwartznyc: BuildPython, DeployPython -- Second phase (following completion of first phase) --- @brianlynn2: DeployExcel --- @tomhealey-icma (TBC): BuildCSharp8, BuildCSharp9, DeployCSharp8, DeployCSharp9
• Support ROTA -- REGnosys will retain responsibility for Level 1 and Level 2 support

Next steps:

• Finalize proposed first five actions: TradeHeader / Apr 19
• Clarify use of secrets: Manuel and Minesh, week of Apr 15
• Implement DAML actions: Chris
• Implement Python actions: Dan
• Propose the next meeting for the week of Apr 22: Dan

[manel-martos]

Contribution improvements agreed on meeting # 4 Apr 29 Source PR

• Resolve snapshot workflow trigger: The build process should run when a PR is created. Discussions around snapshot artifact distribution will be tackled in the future. - Tradeheader
• Maven build action should run under ubuntu image (linux build) - Tradeheader

[dschwartznyc]

Recap and next steps from meeting # 4 - Apr 29, 9 AM EDT, 2 PM BST, 3 PM CEST

Comments and corrections are welcome.

In attendance (by name and GitHub Id or company): Arnau Oller (TradeHeader), Brian Lynn (@brianlynn2), Chris Rayner (@chrisisla), Dan Schwartz (@dschwartznyc), Eleonora Acuna (@eacunaISDA), Hugo Hills (@hugohills-regnosys), Manuel Martos (@manel-martos), Tom Healey (@tomhealey-icma). Apologies: MInesh Patel (@minesh-s-patel)

• TradeHeader has made solid progress on the first 4 actions and the Java build process
• Prototypes completed for three other build actions: C-Sharp (@tomhealey-icma), DAML (@chrisisla), Excel (@brianlynn2)
• Refactoring into language-specific actions is not feasible at this time due to the overhead of ingesting the Rosetta-CDM model

Open question:

• For Snapshots, should the build process trigger when a PR is created or when approved?

Next steps:

• As part of the Governance Review, propose to the SWG that Build Process engineering be delegated to the TAWG (@tomhealey-icma)
• Java build process to leverage Ubuntu image (TradeHeader)
• Finalize C-Sharp (@tomhealey-icma), DAML (@chrisisla), and Excel (@brianlynn2) prototypes and implement actions for the Python build/deploy (@dschwartznyc)
• Suggest regrouping when the above are ready for review

[dschwartznyc]

As discussed in the TAWG meeting on July 11 , due to challenges in engagement this project is on hold.

Under various branches, the repo mentioned above is current with work to date.

The next steps will be to finalize and test the changes based on agreement on how to address the open questions including those listed below:

1. What should trigger the process? Is it a raised PR? A commit/merge? An approved PR? Another action?
2. Should the new process replicate the current one and maintain separate jobs that are executed in parallel. Doing so will require extra action(s) to upload any artifacts that need to be shared. Uploading data leads to a secondary question of whether to clean up when the process completes. Alternatively, to avoid upload, download and clean up we can shift to one big job.

For reference, participants in the project (by name and GitHub Id or company): Arnau Oller (TradeHeader), Brian Lynn (@brianlynn2), Chris Rayner (@chrisisla), Dan Schwartz (@dschwartznyc), Eleonora Acuna (@eacunaISDA), Hugo Hills (@hugohills-regnosys), Manuel Martos (@manel-martos), MInesh Patel (@minesh-s-patel),Tom Healey (@tomhealey-icma).