CDM Technical On-Boarding - FINOS Internal only

[robmoffat]

Contribution process (v. 1.1, last updated on December 20, 2021)

Below is the list of tasks that FINOS Team and the contribution author go through in order to complete the FINOS contribution process.

Identify project meta (Lead: FINOS Contrib POC, Support: FINOS Marketing)

• [x] Project main coordinates
  • [x] Project Name - Common Domain Model
  • [x] Project Slug - common-domain-model
  • [x] Does the name have a registered trademark? yes
    • [x] Work with Ian and Alan
    • [x] Transfer trademark to FINOS/LF
  • [x] Request logo design (if needed) - requested
    • [x] Logo is accepted by stakeholders - https://github.com/finos/finos-landscape/blob/staging/hosted_logos/cdm.svg
• [x] Category and sub-category (for FINOS Landscape) - Data & Business Logic > Models
• [x] Is there existing code? If so, is it public? - yes and no
  • [x] If not, can you grant read access to user github.com/finos-admin ?
• [x] Was the project ever released? If so, are releases public? - yes
  • [x] And what's the latest released version? 2.179.4
• [x] Team composition: lead maintainer and other maintainers
  • [x] Minesh identifies all existing contributors to CDM
• [x] Are public meetings currently held for the project? - yes
  • Every Tuesday, hosted by ISDA
  • More meetings for other tracks
• [x] Are meeting minutes, agenda and attendance tracked? - yes
• [x] Is Continuous Integration used? If so, which system is used? - yes
• [x] Is there an existing Documentation website? If not, would you like to have one? - yes - https://www.isda.org/2019/10/14/isda-common-domain-model/

Maintainers, contributors and CLAs (Lead: FINOS Contrib POC, Support: FINOS infra)

• [x] For each maintainer identified in the previous step, collect: the following info:
  • Fullname
  • GitHub username
  • Corporate email address
• [x] Identify other existing contributors (assuming there's a contribution history (eg Git history)
• [ ] Check if trade associations (and its members) are all onboarded on EasyCLA
• [ ] EasyCLA supports CSL license (on FINOS)
• [x] Engage with FINOS Legal team to figure out what’s needed to cover all maintainers and contributors with FINOS CLA
• [x] Reach out to contributors and employers to coordinate CLA signatures

Project Communication Channel(s)

• [x] Ask maintainers which communications channels they'd like to use
• Asynchronous
  • [x] GitHub Issues (public)
  • [x] Mailing-list
• Synchronous
  • [ ] FINOS Slack Channel (general public Slack / leadership private Slack)
• [x] Link communication channels linked front and center in the project README.md

Code validation (only if code is contributed) (Lead: FINOS Infra)

• The code bindings build process runs successfully. Code bindings have no direct/transitive dependencies
• [x] The codebase doesn’t include any patent or copyright that conflicts with FINOS Governance and bylaws (to be validated with FINOS Legal team)
• Apply project blueprint contents - see docs - will be tackled after transfer, see below

Generated code

• [x] The codebase doesn’t have HIGH or CRITICAL CVEs across direct and transitive libraries
• [x] The codebase doesn’t have any unfriendly licenses across direct and transitive libraries

Approval (Lead: FINOS Infra)

• [x] Assign issue to Executive Director (@mindthegab) to trigger voting
• [x] (optional) if additional socialization is required, the Executive Director may bring projects to the FINOS Governing Board
• [x] FINOS accepts the contribution (and the contribution process can move forward)

Code transfer (Lead: FINOS Infra)

• [x] Minesh to understand how this process will work for members.
• [x] Backup (even with screenshot) GitHub permissions of the repository to transfer
• [x] We need a new name for the project under FINOS (not rosetta-cdm) Ian Suggestion: common-domain-model ACCEPTED
• [x] Check GitHub repository transfer requirements:
  • [x] finos-admin has Admin to all repositories to transfer
  • [x] finos-admin ia allowed to transfer repositories out of the org
  • [x] if the repository is owned by a user (and not an org), the user must be able to transfer the repository to finos-admin
• [x] Review FINOS project blueprint contents
  • [x] Project title/description in README
  • [x] Project badge in README
  • [x] Contributing in README
  • [x] Rob & Mao: Add section about contributing via Legend (add limitations)
  • [ ] CONTRIBUTING.md - For Legal Track - Merge contents from https://cdm.docs.rosetta-technology.io/source/contribution.html#how-to-contribute , https://github.com/finos/rosetta-cdm#contribution-via-rosetta and https://github.com/finos/standards-project-blueprint
  • [ ] License in README - For Legal Track
  • [ ] LICENSE (look for {} placeholders) For Legal Track
• [ ] Check protection settings and disable after transfer if necessary
• [ ] Transfer all code assets as GitHub repositories under github.com/finos
• [ ] Invite GitHub usernames to GitHub FINOS Org
• [ ] Create <project-name>-maintainers GitHub team and invite users
• [ ] Configure finos-admins (Maintain role) and finos-staff (Triage role) team permissions
• [ ] cdm.finos.org redirects to cdm.docs.rosetta-technology.io

Infra setup (Lead: FINOS Infra)

• [x] Enable security vulnerabilities scanning
• [ ] Enable EasyCLA. https://docs.linuxfoundation.org/lfx/easycla
• [ ] Add project to metadata
• [ ] Add identities, orgs and affiliations to metadata (deprecated by EasyCLA)
• [ ] Add logo to FINOS landscape
  • [ ] Create staging branch on finos/finos-landscape
  • [ ] Merge finos/metadata changes on master (will udpdate landscape.yml in finos/finos-landscape)
  • [ ] Create PR from staging branch on finos/finos-landscape
  • [ ] Review Netlify preview
  • [ ] Merge PR
• [ ] Add project maintainers emails to finos-project-maintainers@finos.org list
• [ ] Add project maintainers GitHub usernames to the project-maintainers Team
• [ ] Onboard project on LF systems (SFDC, Insights, EasyCLA, Groups.io)
• [ ] Set groupId to org.finos.cdm
• Set Java artifact deployment to use Maven Central, Python artifact deployment to use Pypi (with finos-admin credentials) and manage all other code binding releases - will happen after contribution

Announcement (Lead: FINOS Contrib POC)

• [ ] Lead maintainer works with FINOS marketing to send out announcement to announce@finos.org , checkout announcement template at the Contribution page
• [ ] Notify FINOS Contrib POC and FINOS marketing (@grizzwolf + finos-marketing internal Slack channel)

[robmoffat]

@minesh-s-patel @iansloyan

[iansloyan]

Yes I have access now, thank you @robmoffat

[minesh-s-patel]

README file is available in this PR: https://github.com/REGnosys/rosetta-cdm/blob/5026e807e1f6889a48fec9ea8cc8832ddf62a4b1/README.md

[robmoffat]

Update for 13 October 2022:

• @minesh-s-patel added @robmoffat & @maoo to the cdm-reviewers team so we can access Regnosis/rosetta-cdm

• @minesh-s-patel has created a README

  • Legal Track:
    • needs to update the readme to talk about contributing and the CLA process (marked this responsibility in the checklist above)
    • needs to fill out the License details (again, legal responsibility to decide this)
  • @robmoffat & @maoo : Add section about contributing via Legend (add limitations)

• We need a new name for the rosetta-cdm repo (added checkbox above)

• CVE Scanning: xText needs to be updated for the CVE scanning to pass - @minesh-s-patel would like to raise an issue for this and do the work (2 weeks effort) in January. This sounded reasonable to me since it wasn't a runtime dependency. @maoo can you comment thanks.

• Discussed how EasyCLA works. I've hooked @minesh-s-patel up with the documentation. He's going to look into how this will work given Rosetta Studio's model of committing on behalf of its users.

[minesh-s-patel]

@robmoffat

The Current CDM Github will have its first stable release (3.0.0) in the next couple of weeks. The 3.0.0 will be the candidate version that will be contributed to FINOS.

Tech tasks in order

• Make Rosetta Testing library available (currently it contains dependencies on private artifacts)
• Once 3.0.0 is released - kick off the process to fork onto finos (Early Nov)
• Perform the repo rename (name tbd)
• Change the group/artifact ids in all of the poms
• Use existing codefresh builds for now - migrate to GHA in the future (Q1 23)
• Start scanning CVEs and create FINOS github issues to resolve
• Migrate documentation to finos domain (from https://cdm.docs.rosetta-technology.io/)
• Support FINOS CDM in Rosetta (work to be done outside of FINOS by the REGnosys product team)
• Document contribution workflow from Rosetta

[robmoffat]

Update 19th October 2022:

From @minesh-s-patel:

The Current CDM Github will have its first stable release (3.0.0) in the next couple of weeks

CDM 3.0.0 on Nov 7th.

From @iansloyan:

• Paper analysis complete
• Sprint ending on 7th nov will include: namespace changes, static data cleanup, removing of copyright assertions over things like "maturity date".

Agreed in the meeting:

• CDM 3.0.0 -> FCDM 4.0.0 (in FINOS). Transferred repo in week of Nov 7th.
• Regnosis will fork (the FINOS repo) in order to support previous production versions.
• CDM's test suite has a dependency on Regnosys' proprietary (but publicly available) test tools. @minesh-s-patel has a plan to open source this, but atm this is not a show-stopper
• We need to raise the EasyCLA process in the steering group for the Legal Track. Also are we using CSL license or FINOS IP Project?

[robmoffat]

From 24 Oct Meeting:

• common-domain-model is the accepted slug name
• We are using CSL License

[robmoffat]

From 28 Oct Meeting:

• can't proceed with EasyCLA onboarding for CDM until we have EasyCLA CSL issues resolved.
• @minesh-s-patel took us through the tasks for the next sprint (starting 7 Nov). Approx 3 days work to complete the transfer.
• Since we have dependency on EasyCLA, work will continue to do all onboarding tasks starting on 7th Nov like so:
  • 3.0.0 - security, refactoring, cleanup -> ready to transfer
  • make the transfer, github.com/finos/cdm is public and avaialable
  • regnosys forks github.com/finos/cdm into github.com/Regnosys/cdm (public or private)
  • Regnosys makes changes into the fork, and releases from there, if/when necessary
  • EasyCLA is sorted
  • Regnosys submits a Pull Request to github.com/finos/cdm

[robmoffat]

Hi @minesh-s-patel,

@maoo and I are looking at trying to complete the code validation steps ahead of the repo transfer later this week. We are running into some build issues both on rosetta-cdm and rosetta-dsl.

First, on rosetta-dsl:

mvn install 

...

[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project com.regnosys.rosetta: Could not resolve dependencies for project com.regnosys.rosetta:com.regnosys.rosetta:eclipse-plugin:0.0.0.master: Could not find artifact org.eclipse.xsemantics:org.eclipse.xsemantics.runtime:jar:1.20.0-SNAPSHOT -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.

On rosetta-cdm:

mvn org.owasp:dependency-check-maven:check -DfailBuildOnCVSS=7     

....

[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project cdm: Could not resolve dependencies for project com.isda:cdm:jar:0.0.0.master: The following artifacts could not be resolved: com.regnosys.rosetta:com.regnosys.rosetta.lib:jar:4.44.2, com.regnosys.rosetta:com.regnosys.rosetta.blueprints:jar:4.44.2, com.regnosys:rosetta-common:jar:3.18.0, com.regnosys:rosetta-testing:jar:3.18.0, com.regnosys:ingest-test-framework:jar:3.18.0: Failure to find com.regnosys.rosetta:com.regnosys.rosetta.lib:jar:4.44.2 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

@maoo tells me there is an issue raised for this already: https://github.com/REGnosys/rosetta-dsl/pull/417 This is the one right?

[robmoffat]

From 9 Nov Meeting:

• Releasing: @minesh-s-patel wants to make sure that after the code transfer to FINOS, releases can begin again quickly. 2 weeks would be fine.
• FINOS Technical Checklist: RegnoSYS have started this work, planned to end on 25th Nov. Simon to work with @maoo to get just the security tasks done for DSL, then CDM (including resolving the above issue about public build-ability)
• Consumption Numbers: @maoo asked for numbers around bindings (scala, java, kotlin, go) to prioritise releases. However, it's an enterprise feature to know downloads, and RegnoSYS don't have this.
• xsemantics issue: RegnoSYS have found the developer to do a proper release of this library into maven central, but this now needs a whole update of rosetta-del dependencies before it can be built publicly.
• @iansloyan has created a PR for removing the private IP from the CDM. Couple of days work on ISDA to merge it. Will happen after the 3.0.0. release which happens today. Not necessarily scheduled immediately, just needs doing before ICMA contribution.

[robmoffat]

Nov 16:

• @minesh-s-patel update on the Finos Technical Checklist:

  • DSL 5.0.0 with the xsemantics and CVE scanning has been released yesterday.
  • The rest of the infra code will start to be updated today
  • The really hard parts affecting downstream app due to the library changes has been resolved and going into UAT by EOD.
  • The CDM scanning is has been done on a branch and is looking very positive.
  • Ian’s PR is with me and it will be ready to merge pending final approval from the TAs.

• Releasing: @minesh-s-patel says:

  • 3.0.0 has been cut, is out now.
  • 4.0.0-dev.0 release is done
  • 4.0.0-dev.(build 1) will contain @iansloyan 's PR. (needs final sign-off from TAs before merge)
  • 4.0.0-dev.(build 2) will contain ICMA's mandatory contribution (no idea on this)
  • these will be contributed to FINOS as version 4.0.0

• Consumption Numbers: @iansloyan says that:

  • (elided) use DRR and are on 3.0.0
  • ISDA Create are using it for DRR and are on version 2.x
  • Some other users who are more sporadic on consumption will wait for a level of stability within FINOS before upgrading.
  • Often consumers see this as a static model, rather than something that should be upgraded.
  • 100's of people and dozens of institutions have accepted the terms and done a download.
  • 500 unique users in last 6 months. @minesh-s-patel : more if you include DRR model.

NB: ICMA contribution is a mandatory dependency. No sight of where this is right now. @iansloyan says it is going through the approval process.

[robmoffat]

23rd Nov:

CVE Scanning

• CVE Scanning for rosetta-dsl and cdm are done. rosetta-dsl dev release completed with "cleaned up" dependencies. Outputs from code generations of CDM now free of CVEs. ✅
• @maoo to validate this.
• Suggestion to add a badge on the README file to indicate this.

Independent Build

• @minesh-s-patel dependencies in artifactory still preventing mvn clean install.
• There is a piece of code that generates example data in /test for users to look at. It also exercises the model by generating this data. It is not a part of the built cdm but is required for a build.
• @minesh-s-patel is moving this out of the main mvn build, it's not on the critical path and can be tidied up later.
• @minesh-s-patel will detail in a following comment further libraries that they will be open-sourcing that currently are non-runtime dependencies.
• He's doing this work during the current sprint (finishing 2nd Dec)

@iansloyan 's PR:

• Has been reviewed, but not signed off by TAs. Back with Ian.
• Waiting on the ICMA submission since a dependency now exists between the two. Hasn't been approved yet, in process. "Might take a little longer".
• 4.0.0-dev.(build 1) will now wait for the ICMA contribution.

[robmoffat]

30th Nov:

Independent Build

• mvn clean install - no closed deps / bad licenses (apart from test mentioned above, but this is acceptable)
• rosetta-testing, rosetta-common now open-sourced.
• repositories in pom.xml uses the RegnoSYS artifactory setting

CVE Scanning

• Added to the FINOS Fork of CDM and CVEs.
• 3 CVEs found. @minesh-s-patel to investigate/bump version numbers
• @minesh-s-patel to merge PR into main CDM project: https://github.com/REGnosys/rosetta-cdm/pull/1847

LIcense Scanning

• Added to the FINOS Fork of CDM.
• Currently failing in GitHub.
• @minesh-s-patel to investigate / fix up

@iansloyan 's PR

• TA review: all said it's ok.
• Meeting on Friday to review ICMA contribution

[minesh-s-patel]

The GitHub Actions PR cannot be merged because the REGnosys CDM Repo does not support github actions (CDM project: https://github.com/REGnosys/rosetta-cdm/pull/1847). This PR can be merged once migrated to the open source finos org.

[minesh-s-patel]

@robmoffat

I ran the following command (from your github actions PR) and the build was successful

What was the issue you was seeing?

mvn org.codehaus.mojo:license-maven-plugin:2.0.0:download-licenses

[minesh-s-patel]

I think we have a clean run for licences. @maoo @robmoffat CAn you confirm when you have a sec?

> export ALLOW_LICENSES="'The Apache Software License, Version 2.0' and licenses/license/name!='BSD' and licenses/license/name!='BSD-style license' and licenses/license/name!='Apache License, Version 2.0'"

> export REPORT_PATH="target/generated-resources/licenses.xml"

> mvn org.codehaus.mojo:license-maven-plugin:2.0.0:download-licenses
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for cdm-parent 0.0.0.master:
[INFO]
[INFO] cdm-parent ......................................... SUCCESS [  1.283 s]
[INFO] rosetta-source ..................................... SUCCESS [ 11.508 s]
[INFO] scheme-import ...................................... SUCCESS [  7.329 s]
[INFO] rosetta-project .................................... SUCCESS [  3.068 s]
[INFO] tests .............................................. SUCCESS [  1.698 s]
[INFO] isda-demo .......................................... SUCCESS [  0.195 s]
[INFO] cdm-distribution ................................... SUCCESS [  0.201 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  25.544 s
[INFO] Finished at: 2022-12-13T09:42:45Z
[INFO] ------------------------------------------------------------------------

> xq "//dependency[licenses/license/name!=$ALLOW_LICENSES]" ./$REPORT_PATH
<results/>

[minesh-s-patel]

@robmoffat

Can you please mark the following as checked?

Logo is accepted by stakeholders (We used the logo at OSFF - so we can mark that one off) Minesh identifies all (future) contributors to CDM The code bindings build process runs successfully. Status: waiting on refactor to new namespaces / structure > 7th Nov The codebase doesn’t have any unfriendly licenses across direct and transitive libraries

Only thing on my list:

The codebase doesn’t have HIGH or CRITICAL CVEs across direct and transitive libraries

This is incorrectly marked as done.

Rob & Mao: Add section about contributing via Legend (add limitations)

Most of the other actions are either for ISDA/ISLA/ICMA/FINOS or require information in order to complete (e.g. what to put in the NOTICE file?)

[maoo]

CVE and license scanning are green on Rosetta CDM. Well done @minesh-s-patel !

https://github.com/finos/rosetta-cdm/actions/runs/3684917909 https://github.com/finos/rosetta-cdm/actions/runs/3684696554

[robmoffat]

14th Dec:

Main areas still to resolve prior to onboarding:

• Proper Contribution Issue & TSC Approval of CDM
• EasyCLA - we need David Deal's changes completed, we can do a demo to all of the main contributors to show how to get onboarded
• NOTICE file - we need legal to give the copy for this.
• CONTRIBUTING.md - legal track need to review and approve this.

After Onboarding:

• DNS Redirect cdm.finos.org to existing docs
• Slack Channel (possibly ok as-is)
• LICENSE: as soon as it's added to FINOS
• Create GitHub action for sphinx deploy into cdm.finos.org GitHub pages, and re-point links. (Assign to RegnoSYS)
• Fix Maven Central deployment of generated artifacts

[robmoffat]

https://jira.linuxfoundation.org/plugins/servlet/desk/portal/4/SUPPORT-14860

[maoo]

Just moved all checks into https://github.com/finos/community/issues/224 . Closing this issue as dup.