search

finos / community

FINOS Community, Project and SIG wide collaboration space
http://community.finos.org
64 stars 28 forks source link

JupyterLab Templates - Software Project Contribution and Onboarding #215

Closed timkpaine closed 7 months ago

timkpaine commented 1 year ago

JupyterLab Templates - Software Project Contribution and Onboarding

I'd like to propose a new track for FINOS around Jupyter Notebooks and their related tools. As Jupyter has become the de-facto standard tool for data science around the world, it has also found its way into the hands of buy and sell side business analysts, traders, and quantitative researchers. Coding has become an essential tool for finance and has begun to unseat excel as the data analysis tool of choice for the industry.

Additionally, as more and more financial organizations begin to explore open source, Jupyter ecosystem tooling has become a popular avenue for participation. Here is a brief survey of the open source presence of some buy and sell side institutions, and one can see the presence of numerous Jupyter ecosystem projects:

Additionally, it is my understanding that JP Morgan, Two Sigma, and Bloomberg have employed Jupyter core maintainers as employees, and numerous other financial organizations have contributed directly to Jupyter projects.

Currently, I maintain 4 Jupyter-based projects under the JP Morgan GitHub organization, as well as Perspective under the FINOS organization (which also features a Jupyter widget as a core aspect of its functionality). These projects are currently being actively used by many financial institutions and FINOS member organizations, and I believe the establishment of a new track for Jupyter ecosystem projects will help foster wider collaboration. I have several projects ready for incubation under this new track should it be adopted.

As an inaugural project, I’d like to propose the JupyterLab Templates project. This is a relatively established and widely used project that would benefit from incubation in FINOS, and allow for further collaboration across the open source member ecosystem.

Business Problem

Jupyter notebooks are the de facto standard tool for data scientists, quantitative researchers, and business analysts. Often times, they want to work from standard project templates. As an example, a risk quant might want a set of standard notebooks that they can quickly create, fill in a few details, and run. An analysts might want a “company profile” notebook they can quickly instantiate, fill in a company name, and generate a report. The first building block of these is the ability to package and distribute template notebooks, which is the functionality provided by the JupyterLab Templates library.

Solution

The JupyterLab Templates file lets you collect and install a collection of notebooks as part of your python (or other) package. It then integrates with the JupyterLab user interface to provide a new notebook card called “Template Notebook”. Clicking on this provides a browser which the user can use to select amongst any of the preinstalled notebook templates.

Roadmap

There is a tentative roadmap in the project issues (https://github.com/jpmorganchase/jupyterlab_templates/issues). Immediate goals for the project are to reimplement the UI to be more intuitive and allow for better searching/filtering, to allow for bundled/tarball/git remote templates, integrate a notebook template with its runnable environment for reproducibility, and evangelize the notebook templates project to encourage greater participation amongst open source projects.

Current State

Project is actively maintained by at least 2 employees of FINOS members. Project has 300 stars and ~9000 monthly downloads.

Existing materials

https://github.com/jpmorganchase/jupyterlab_templates

Development team

@timkpaine @vidartf (here and there) @neilslinger (oversight)

Contributor Commitment

@timkpaine will actively maintain this project for the foreseeable future, with collaboration from JP Morgan.

Confirmed contributors

@timkpaine / @Point72 / timothy.paine@cubistsystematic.com

Target Contributors

Any and all, including those not affiliated with FINOS or involved in finance (I regularly receive feedback via email from academic and non-financial industry users).

Project Communication Channel(s)

Just GitHub issues

Infrastructure needs

None, just GitHub

Understanding FINOS Onboarding Requirements

As a project onboarding into FINOS, you will need to familiarize yourself and your contributor team with the following materials:

2. FINOS Approval

Below is the list of tasks that the FINOS Team go through in order to complete the FINOS approval process. At this point, a FINOS Point-of-Contact (POC) should be assigned to this GitHub issue.

Please do not edit these contents when completing part 1, "describing the contribution" above.

Record The Contribution

Kick-off meeting

FINOS Contrib POC

Proposal (Lead Maintainer)

Technical Steering Committee Approval

TSC Findings / Report

TSC to enter findings summary here.

3. Preparing For Onboarding

Before the FINOS team can onboard your project, there are a few housekeeping that need to be taken care of. These must be completed by the contributor, with help if required from the POC or FINOS Infra.

Logo / Trademarks

Coding Standards

FINOS Project Standards

Add documentation here

4. FINOS Onboarding

This is performed by FINOS Infra once the three previous stages are complete, with support from the contributor and the FINOS POC.

Maintainers, Contributors and CLAs

Validation (only if code is contributed)

Code transfer

Project Communication Channel(s)

Infra setup

5. Announcement

(Lead: FINOS Contrib POC)

DISCLAIMER: This e-mail message and any attachments are intended solely for the use of the individual or entity to which it is addressed and may contain information that is confidential or legally privileged. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, copying or other use of this message or its attachments is strictly prohibited. If you have received this message in error, please notify the sender immediately and permanently delete this message and any attachments.

maoo commented 1 year ago

Thanks for this @timkpaine , it seems a very interesting contribution! We'll start working on this and reach out to organize a demo to our Technical Steering Committee. Apologies for the late response, we're very busy wrapping all up for https://www.finos.org/hosted-events/2022-12-08-open-source-in-finance-forum-new-york

jonfreedman commented 1 year ago

@ColinEberhardt not sure what you think but seems like a demo of this would be overkill. I think the relevant contribution principles are also satisfied based on interest from two member firms. @mindthegab @jgavronsky I don't know whether the proposal of a new "track" is part of the TSC remit or not?

robmoffat commented 1 year ago

@ColinEberhardt and I are going to discuss with @mindthegab and @jgavronsky tomorrow... will get back to you @jonfreedman

mindthegab commented 1 year ago

@jonfreedman @ColinEberhardt @finos/technical-steering-committee

I'd like to get a formal +1 from the TSC to at least accept the contribution of the https://github.com/jpmorganchase/jupyterlab_templates and then after contribution we can discuss the creation of a new SIG (Special Interest Group) which I believe would be the closest to the concept of "track" that @timkpaine has proposed.

We have some degree of urgency here (happy to chat 1:1) to complete this before the holidays, so also happy to work by slient consensus given there's already a general support for the project (2 firms involved, mature, etc).

Can we have a couple of +1s here?

maoo commented 11 months ago

Just checked (again) CVE and license scanning, all good (see below)

NodeJS

# Install production dependencies from the lockfile
cd js ; yarn install --frozen-lockfile --production

# License scanning; all identified licenses are compatible with (outbound) Apache 2.0 license
node-license-validator . --allow-licenses Apache-2.0 MIT BSD-2-Clause BSD-3-Clause ISC

# CVE scanning, no CVEs found
npx --yes auditjs ossi

Python

# Generate requirements.txt and install scanning libs
poetry export --without-hashes -f requirements.txt --output requirements.txt
pip install safety pip-licenses

# CVE scanning,  0 vulnerabilities found, 0 vulnerabilities ignored
safety check --full-report -r requirements.txt

# License scanning; all identified licenses are compatible with (outbound) Apache 2.0 license
pip-licenses --allow-only="MIT License; BSD License; Apache Software License; Mozilla Public License 2.0 (MPL 2.0)"

I also noticed that Makefile facilitates the use of semgrep, but it's not invoked by any Github Action, as far as I see ; it could be helpful to run Semgrep periodically and on code changes via Github Actions.

maoo commented 11 months ago

This is a recap of what's missing - @TheJuanAndOnly99 , could you please work on the following items and submit a Pull Request?

psmulovics commented 11 months ago

Reviewed item as TOC member, I see no issues with progressing forward.

TheJuanAndOnly99 commented 11 months ago

Hey @psmulovics thanks for the comment. We've actually stumbled into a bit of a "dispute" as the JupyterLab Templates maintainers believe that adding the @finos scope to the package name would be disruptive as it is already a popular/widely used package. We are going to bring this up to the FINOS TOC at the next meeting for resolution.

psmulovics commented 11 months ago

It has been allowed in some other cases as well to miss the scope and use tags, a good example is https://www.nuget.org/packages?q=Tags%3A%22finos%22 - only one package has the actual @finos scope in the name, the rest is using it as a tag.

TheJuanAndOnly99 commented 7 months ago

All contribution activities have been completed. We can safely close this issue. Congratulations to the team!