search

finos / community

FINOS Community, Project and SIG wide collaboration space
http://community.finos.org
65 stars 28 forks source link

Desktop Agent Bridging - Software Project Contribution and Onboarding #204

Closed Vivek-NatWest closed 1 year ago

Vivek-NatWest commented 2 years ago

Onboarding Process

Completing an onboarding of a project into FINOS requires following these 5 main steps:

  1. Describing the Contribution led by contributor
  2. FINOS Approval led by FINOS Point of Contact (POC)
  3. Preparing for Onboarding led by contributor
  4. Onboarding completed by FINOS Infra
  5. Announcement led by FINOS Point of Contact (POC)

1. Describing The Contribution

This is a list of questions that need to be answered by the contributor in order to allow a new project to pass to the approval stage of onboarding.

Business Problem

FDC3 implementations (aka Desktop Agents) only facilitate interoperation between applications running within a single context, thus limiting user workflows to the boundary of the Desktop Agent.

To facilitate complex workflows across different platforms requires applications running on one Desktop Agent to integrate with FDC3 applications running on one or more other Desktop Agents for the same user.

Additionally it is desirable to enable legacy standalone applications to participate in FDC3 workflows via the sending and receiving of broadcast context.

Finally, due to performance constraints, many users choose to run multiple desktop machines and therefore Desktop Agents and Standalone Applications may be running on multiple different hosts for a given user but still need to participate in a shared FDC3 workflow.

There is therefore a need for bridging of messages between Desktop Agents as identified in this issue and associated discussion group.

https://github.com/finos/FDC3/discussions/544 https://github.com/finos/FDC3/issues/798

Proposed Solution

In response to the issue we have developed a number of components and patterns that enable Desktop Agent Bridging across multiple hosts:

Backplane: The Desktop Agent Bridge: This is a headless process which runs in background on user's desktop allowing desktop agents to connect and communicate with each other through web socket connections. Each desktop has Backplane running locally and they communicate with each other across network boundaries over HTTP. Discovery of member Backplanes is subject to implementation of the discovery interface.

Backplane JS Client: A Client library to connect and communicate with Backplane from JS applications and JS Desktop Agent implementations e.g. Finsemble, OpenFin

Backplane .NET Client: A Client library to connect and communicate with Backplane from .NET Applications. Enables standalone .NET applications to participate in FDC3 workflow by sending and receiving broadcast context.

Finsemble-Backplane Service: Code snippet of Finsemble service facilitating connection and communication between the Finsemble Desktop Agent and Backplane. This leverages the Backplane JS Client.

Tentative Roadmap

The goal of this project is to widen the horizon of FDC3 based interop and ease its adoption by bridging desktop agents and allowing workflows to span across desktops.

Future development targets are to:

Current State

In current state, all the components mentioned in proposed solution are available and they support broadcast context based on FDC3 specs.

Existing Materials

If materials already exist, provide a link to them that Foundation staff can access - if it's in a private GitHub.com repositories, you should invite the finos-admin user with R/O permissions to those repositories

Development Team

Maintainers

Name Affiliation Work Email Address Github / GitLab username
Vivek Kumar Giri Natwest Markets vivek.kumargiri@natwestmarkets.com Vivek-Natwest
Aaron Haines Natwest Markets aaron.haines@natwest.com Aaron-Haines
Manish Bhutani Natwest Markets Manish.Bhutani@natwestmarkets.com Manish-NWM

Confirmed contributors

Name Affiliation Work Email Address Github / GitLab username
Kris West Cosaic kris@cosaic.io kriswest
Tiago Pina Cosaic tiago.pina@cosaic.io tpina

Target Contributors

Members of Desktop Agent Bridging discussion group.

Project Communication Channel(s)

Understanding FINOS Onboarding Requirements

As a project onboarding into FINOS, you will need to familiarize yourself and your contributor team with the following materials:

2. FINOS Approval

Below is the list of tasks that the FINOS Team go through in order to complete the FINOS approval process. At this point, a FINOS Point-of-Contact (POC) should be assigned to this GitHub issue.

Please do not edit these contents when completing part 1, "describing the contribution" above.

Record The Contribution

Kick-off meeting

FINOS Contrib POC

Proposal (Lead Maintainer)

Technical Steering Committee Approval

TSC Findings / Report

See @ColinEberhardt's comment below

3. Preparing For Onboarding

Before the FINOS team can onboard your project, there are a few housekeeping that need to be taken care of. These must be completed by the contributor, with help if required from the POC or FINOS Infra.

Logo / Trademarks

Coding Standards

FINOS Project Standards

Add documentation here

4. FINOS Onboarding

This is performed by FINOS Infra once the three previous stages are complete, with support from the contributor and the FINOS POC.

Maintainers, Contributors and CLAs

Validation (only if code is contributed)

Code transfer

Project Communication Channel(s)

Infra setup

5. Announcement

(Lead: FINOS Contrib POC)

kriswest commented 2 years ago

I strongly support this proposed contribution to FINOS.

The proposed Desktop Agent Bridging Part of the FDC3 standard requires a separate piece of software ('the bridge') to enable connections between FDC3 Desktop Agents:

image

Both Natwest Market's practical experience of developing and working with the backplane and the contribution of its current codebase will help ensure that a full implementation of the proposed bridge will be possible in a much shorter timescale and that that implementation is Open Source and can be contributed to by the wider community (including banks, asset managers or consultants looking to use it as well as Desktop Agent vendors that will be working with it). As the bridge is intended to interconnect Desktop Agents (open source, in-house at financial services orgs and those from commercial vendors), the development of that bridge implementation in the open is by far the best approach.

mattjamieson commented 2 years ago

I agree entirely with @kriswest's comment above and I'm excited to see this contribution from NatWest which will significantly accelerate the adoption of FDC3 desktop agent bridging.

mcleo-d commented 2 years ago

This is to confirm this project was demoed to the FDC3 Desktop Bridging working group on 28th September 2022 by @Vivek-NatWest - https://github.com/finos/FDC3/issues/822

mcleo-d commented 2 years ago

@maoo and @robmoffat - @Vivek-NatWest has requested help understanding the best practise to setup contributions between the FINOS GitHub Organisation and an internal NatWest repository. Can you take this conversation forward with the FINOS infra team? Many thanks.

mcleo-d commented 2 years ago

On the back of the project demo at https://github.com/finos/FDC3/issues/822, I have applied the ready-for-tsc label and have assigned this item to @kriswest and @ColinEberhardt to take through the FINOS TSC decision making process.

cc @Vivek-NatWest

mindthegab commented 2 years ago

Thanks everyone involved for the progress here. Just to pull all the threads together, for the benefit of @Vivek-NatWest and the contributing team:

  1. The TSC (@ColinEberhardt) has asked to the FDC3 Maintainers (@finos/fdc3-maintainers) to advise on this contribution
  2. Based on the demo that happened last week (https://github.com/finos/FDC3/issues/822), FDC3 maintainers are meeting next week on Wed to opine on this

@kriswest can I ask you to post findings back here once you have a group consensus, so @ColinEberhardt and I can action in either direction?

Thanks!

kriswest commented 2 years ago

@mindthegab sure, will do.

ColinEberhardt commented 2 years ago

The following is a summary of the TSC’s assessment of the Desktop Agent Bridge contribution, evaluating this project with respect to our contribution principles. This has been undertaken in close collaboration with the FDC3 maintainers.

Background

Both the TSC and FDC3 maintainers advise that the FINOS Team and Governing Board accept this contribution.

Evaluation

Note: For the purposes of this evaluation, the contribution (an Implementation of the proposed Desktop Agent Bridging addition to the FDC3 Standard) will be referred to as 'Backplane'.

Contributions must be aligned with the purpose of FINOS

The contribution is intended to provide infrastructure (Known as a Desktop Agent Bridge or DAB) that can interconnect Desktop Agents (DAs), that implement the FDC3 standard, according to a proposed addition to the standard (currently approaching a complete draft).

The interconnect of DAs solves for a problem/use case that has been expressed by multiple participants in FDC3 from Financial Services organisations and vendors. The Desktop Agent Bridging Discussion group, which is overseeing the drafting of the proposed FDC3 Standard addition, has been effective, including representatives from a cross-section of the FDC3/Financial services community (including Banks, Asset Managers, DA vendors, Terminal vendors and Application vendors).

Contributing a (project to implement a) DAB promotes collaboration in its implementation and (through the inclusion of client code to connect to and communicate with the DAB) simplifies the implementation of support for the proposed addition to FDC3 for DA vendors and other implementors of the FDC3 Standard.

As such, the consensus of the FDC3 maintainers and TSC is that the contribution meets this principle,

A proposed standard must not compete with other FINOS standards

The consensus among the FDC3 maintainers and TSC is that Backplane is a project and not a standard, therefore, this principle doesn't fully apply.

However, it should be noted that this contribution is intended to implement the draft addition to the FDC3 Standard for Desktop Agent Bridging. As long as the project remains aligned with the FDC3 standard, we believe that this principle will remain satisfied.

An open-source project that is a component of a commercial offering must provide tangible value in its own right

The consensus among the FDC3 maintainers and TSC is that the Backplane is not a part of a commercial offering and will be made freely available to the FDC3 community through FINOS. It does not require the use of any commercial offering.

Further, the Backplane is intended to enable connection/collaboration between implementations of the FDC3 Standard by both commercial vendors and other implementations (including open source implementations) of that Standard, through the use of a separate piece of infrastructure that is not owned by, nor has to be provided by, one of those implementations. Hence, it will provide value in its own right.

Contributions should generate interest within the member organisation

Based on:

The consensus among the FDC3 maintainers and TSC is that there is clear interest in this project to implement the FDC3 standard addition and hence that this principle is satisfied.

Contributions should be projects that deliver long-term value, with a team that supports this goal

The contributors (NatWest Markets) have stated that they already use the Backplane in their production environment. Another bank has stated an interest in participating and rolling out the result of collaboration, according to a specified target timeline, and at least one DA vendor is also committed to using and participating in maintenance.

Further, the Backplane is intended to provide an addition to desktop infrastructure that would be used on a day-to-day basis alongside the FDC3 standard, rather than to solve a short-term problem.

However, the long-term value will be dependent on adoption by DA vendors - the project aims to ease adoption by providing client code to use the bridge. By providing client code for DAs to use when working with a DAB implementation the projects also make it easier for DA vendors to implement their parts of the DAB proposal and should invite collaboration from DA vendors.

It is the consensus of the FDC3 maintainers and TSC that this project should deliver long-term value, has a strong chance of recruiting maintainers and contributors that support its goals and hence that this principle is satisfied.

A proposed contribution that competes with an existing FINOS project should consider potential routes to a merge

The contribution implements a proposed addition to the FDC3 Standard and does not compete with any existing project known to the FDC3 maintainers. Hence, it is the consensus of the FDC3 maintainers and TSC that this principle is satisfied.

However, it should be noted that this contribution is intended to implement the draft addition to the FDC3 Standard for Desktop Agent Bridging. As long as the project remains aligned with the FDC3 standard, we believe that this principle will remain satisfied.

robmoffat commented 1 year ago

@maoo - security scanning doesn't yet include C# tooling, so the team have added CodeQL and "Dependency Review" (mentioned in SecurityScanning). All good?

robmoffat commented 1 year ago

https://github.com/bankofapis/backplane

maoo commented 1 year ago

@maoo - security scanning doesn't yet include C# tooling, so the team have added CodeQL and "Dependency Review" (mentioned in SecurityScanning). All good?

It's ok to use CodeQL and other tools, but I'd like to confirm that:

I tried to read the logs on https://github.com/bankofapis/backplane/actions/workflows/codeql.yml but couldn't find anything related to dependencies linked above.

maoo commented 1 year ago

@maoo - security scanning doesn't yet include C# tooling, so the team have added CodeQL and "Dependency Review" (mentioned in SecurityScanning). All good?

I've seen the results of the scanning, however, I see that the project contains several (I think 3) .csproj files, and a package.json file with runtime dependencies defined in it, which seem to be not scanned by the Github Actions.

Based on this list of dependencies, we'd also need to run some license scanning, so it would be great to have the list of dependecies somewhere; for NodeJS we can automate the license scanning (using node-license-validator, working on it right now), but for csproj files we'll probably have to manually check libraries one by one (we'd take care of it, assuming we have a final list of dependencies).

Thanks! /cc @robmoffat @Vivek-NatWest

Vivek-NatWest commented 1 year ago

@maoo - security scanning doesn't yet include C# tooling, so the team have added CodeQL and "Dependency Review" (mentioned in SecurityScanning). All good?

I've seen the results of the scanning, however, I see that the project contains several (I think 3) .csproj files, and a package.json file with runtime dependencies defined in it, which seem to be not scanned by the Github Actions.

Based on this list of dependencies, we'd also need to run some license scanning, so it would be great to have the list of dependecies somewhere; for NodeJS we can automate the license scanning (using node-license-validator, working on it right now), but for csproj files we'll probably have to manually check libraries one by one (we'd take care of it, assuming we have a final list of dependencies).

Thanks! /cc @robmoffat @Vivek-NatWest

Here is dependency list: https://github.com/bankofapis/backplane/network/dependencies

maoo commented 1 year ago

Here is dependency list: https://github.com/bankofapis/backplane/network/dependencies

Thanks, that helps for the license scanning; we'll work on that.

As per CVEs, I assume that you're using the GitHub Security Advisories right? Given that the list is empty, that means that no CVEs were spotted?

Vivek-NatWest commented 1 year ago

Here is dependency list: https://github.com/bankofapis/backplane/network/dependencies

Thanks, that helps for the license scanning; we'll work on that.

As per CVEs, I assume that you're using the GitHub Security Advisories right? Given that the list is empty, that means that no CVEs were spotted?

Yes.

maoo commented 1 year ago

I've run AuditJS on the Client JS project, and found a CVE. I think that CodeQL is not taking in consideration transitive dependencies.

For C#, since we don't have alternatives yet, we'll have to go with CodeQL, but for NodeJS, I'd suggest running AuditJS, as shown on https://github.com/maoo/backplane/blob/master/.github/workflows/node-cve-scanning.yml

I'm happy to submit the PR; I was reading on https://stackoverflow.com/questions/56634474/npm-how-to-update-upgrade-transitive-dependencies some ways to use a new version of node-fetch (which is coming from @microsoft/signalr), but couldn't succeed yet.

WDYT @Vivek-NatWest ?

Vivek-NatWest commented 1 year ago

I've run AuditJS on the Client JS project, and found a CVE. I think that CodeQL is not taking in consideration transitive dependencies.

For C#, since we don't have alternatives yet, we'll have to go with CodeQL, but for NodeJS, I'd suggest running AuditJS, as shown on https://github.com/maoo/backplane/blob/master/.github/workflows/node-cve-scanning.yml

I'm happy to submit the PR; I was reading on https://stackoverflow.com/questions/56634474/npm-how-to-update-upgrade-transitive-dependencies some ways to use a new version of node-fetch (which is coming from @microsoft/signalr), but couldn't succeed yet.

WDYT @Vivek-NatWest ?

Sure thanks 👍

maoo commented 1 year ago

@Vivek-NatWest - here's the PR https://github.com/bankofapis/backplane/pull/9

robmoffat commented 1 year ago

I was unaware that CodeQL examined dependencies at all - I thought it just looked at the codebase?

Vivek-NatWest commented 1 year ago

I was unaware that CodeQL examined dependencies at all - I thought it just looked at the codebase?

Yes, as per it's documentation, it scans code for vulnerability. Dependency CVE scanning has to be done by some other way. OWASP dependency check for .net, if some action can be created over it. Need to check.

maoo commented 1 year ago

From https://devblogs.microsoft.com/nuget/how-to-scan-nuget-packages-for-security-vulnerabilities/#dotnet-cli , it seems we could simply use the dotnet command. I was able to find some CVEs by running the following commands:

cd src/Finos.Fdc3.Backplane
dotnet restore
dotnet list package --vulnerable --include-transitive

Could you please check on your end and see if there's a way to address the CVEs found?

Vivek-NatWest commented 1 year ago

From https://devblogs.microsoft.com/nuget/how-to-scan-nuget-packages-for-security-vulnerabilities/#dotnet-cli , it seems we could simply use the dotnet command. I was able to find some CVEs by running the following commands:

cd src/Finos.Fdc3.Backplane
dotnet restore
dotnet list package --vulnerable --include-transitive

Could you please check on your end and see if there's a way to address the CVEs found?

Sure, would check and update. Thanks

robmoffat commented 1 year ago

Hi @Vivek-NatWest,

One Minor thing:

thank you!

Vivek-NatWest commented 1 year ago

Removed the reported CVE dependency. CVE passing now for .NET

Vivek-NatWest commented 1 year ago

Hi @Vivek-NatWest,

One Minor thing:

thank you!

Done. Thanks Rob!

maoo commented 1 year ago

Removed the reported CVE dependency. CVE passing now for .NET

I still see the scan failing on https://github.com/maoo/backplane/actions/runs/3480480821/jobs/5820337949#step:5:11

Vivek-NatWest commented 1 year ago

Removed the reported CVE dependency. CVE passing now for .NET

I still see the scan failing on https://github.com/maoo/backplane/actions/runs/3480480821/jobs/5820337949#step:5:11

Here is scan results on master : https://github.com/bankofapis/backplane/actions/runs/3489048483 which is passing. Both transitive dependencies reported are no longer there. Let me know if you observe otherwise.

robmoffat commented 1 year ago

2022 FDC3 Backplane.zip

maoo commented 1 year ago

Congratulations @Vivek-NatWest for the contribution, closing issue.