search

ipfs / devgrants

The IPFS Grant platform connects funding organizations with builders and researchers in the IPFS community.
167 stars 75 forks source link

Open Grant Proposal: JSPM IPFS Integration #280

Closed guybedford closed 1 year ago

guybedford commented 1 year ago

Open Grant Proposal: Universal Optimized Dynamic Module Linking with IPFS, Import Maps & JSPM

Name of Project: JSPM IPFS Integration

Proposal Category: integration-adoption

Proposer: Guy Bedford on behalf of the JSPM Foundation

(Optional) Technical Sponsor: If you have previously discussed this project with a member of the IPFS or Filecoin project teams and they have agreed to be a technical sponsor, include their name and/or github handle here

Do you agree to open source all work you do on behalf of this RFP and dual-license under MIT, APACHE2, or GPL licenses?: Yes

Project Description

JSPM Project Background

The JSPM JavaScript ES module CDN has been running for almost 10 years, and serves more than 3.5TB of optimized executable JavaScript directly to the browser with over 600m+ JavaScript module requests to 10m+ unique IPs per month.

JSPM uses the relatively new web browser feature, Import Maps, to directly map dependencies on the npm package ecosystem to their optimized edge-cached versions on the JSPM CDN, working directly with the native ES module loader in the browser. In this way JavaScript becomes not only a dynamically executed language, but also a dynamically linked language against universal executable URLs at runtime. The benefit of this approach is that instead of browsers downloading multi-megabyte bundles of JS that often contain 90% duplications of existing dependency code, shared dependency code loading can be maintained through localized edge caching directly to the browser.

Project Proposal

The JSPM CDN uses unique URLs with far-future expires for all requests, and previously support was added to the JSPM generator import map generation system to allow generating import maps directly from code hosted on IPFS URLs (https://github.com/jspm/generator/pull/79).

Extending this technique to supporting npm dependencies being loaded directly from IPFS shows promise as a future CDN model, since JSPM is a push-based package builder and not pull-based, there is a clear technical congruency on this path for the current JSPM CDN to migrate from being a centralized CDN system to become a fully decentralized universal module delivery system on top of Filecoin and IPFS.

The project proposal would be to create a version of the JSPM CDN npm package builder and CDN codebase that can also serve as a Filecoin and IPFS service based on integrating this support into the existing CDN codebase that is already running the JSPM CDN today. Further, the proposal is to set up this node infrastructure to be fully decentralized such that anyone could run a distributed node in the system, with optional sharding rules for contributing to only a subset of the 2m+ packages available on npm for smaller nodes.

The proposed project would therefore, as a side effect, involve fully open sourcing the JSPM builder code base, an intricate job given the nature of the project’s development over many years. The combined announcement would be a major milestone for the project as a whole.

This vision would only fully come into its own with support for IPFS and IPFS URLS in import maps in browsers (dependent on having “ipfs://” listed as a “fetch scheme” in the browser). In the mean time, the project would rely on demonstrating the use of IPFS CDN mirrors, server-side workflows via Node.js / Deno / Bun IPFS URL support, and in custom browser builds with direct IPFS support, fully implementing and sharing these workflows as deliverables of the project.

Value

The benefit of the project would be proving out an entirely fundamentally new model of distributed link-time execution with shared edge caching without the need for a centralized CDN. The JSPM CDN as it exists today is a centralized service and point of trust. While it does support a multi-CDN model for users choosing to use any CDN provider, ideally hash-integrity should be maintained for all requests, based on an initial user verification phase during import map creation. IPFS is ideally and uniquely suited to this role.

One risk might be failing to generate enough excitement and providing a poor implementation - if execution is not done to a high enough standard, and users appreciate but decide the concept is not good enough in practise, then that could lead to a bad impression of import maps and this type of dynamic linking approach resulting in users sticking to more traditional CDN approaches instead.

The project is made difficult due to the lack of browser support for IPFS URLs. As described in the proposal section, we can provide demonstrations through custom browser implementations and instrumentation of Node.js with import maps support. The benefit here might then be driving awareness and in turn specification work and adoption in browsers.

Deliverables

The project can be divided into three main parts - the distributed node architecture, the import map generation support, and the workflow demo creation and marketing of the concept.

  1. JSPM CDN Distributed Node with an IPFS “flag”
    In order to get optimized npm dependencies on IPFS, this would be a new open source project, open sourcing the “JSPM package builder” in a way that anyone can run a distributed node. The node would have configuration options for sharding or push or pull-based builds. In addition it could run a traditional server or it could be passed an IPFS flag to run an IPFS Node serving the same package content over IPFS. This forms the bulk of the project as a large infrastructure integration, including getting the open source project to a full release-ready state including comprehensive testing, polishing and documentation.

  2. Implement IPFS as a “provider layer” in the JSPM Generator project.
    The JSPM Generator project is the open source tool that creates import maps against the JSPM CDN. It implements a distributed version of the Node.js module resolution algorithm, and was created by Guy Bedford who was one of the main developers of the Node.js modern module resolution algorithm. While the JSPM Generator project already supports the generation of import maps against code hosted on IPFS URLs, these import maps themselves still reference the JSPM CDN. The deliverable would thus be to add support for the generation of import maps that reference dependencies on IPFS URLs.

  3. Workflow Demos
    Workflow demos will be constructed to show:

    • Import Maps + IPFS executing on Node.js, Deno or Bun. Possibly also on a cloud edge like Fastly Compute@Edge (on which Guy Bedford works full time).
    • Import Maps + IPFS running via an IPFS https service.
    • Import Maps + IPFS running on a custom browser supporting IPFS URLs.
    • Putting together demonstrations of these workflows is very much an important deliverable. Guy Paterson-Jones is also interested in applying to conferences to share this work. This forms the major deliverable in sharing and getting users excited about this novel work and direction.

Development Roadmap

Month 1 - Setting up the Distributed CDN Package Builder Project Structure

  1. Extract the existing JSPM CDN tests into a new open source project, and set up the test harness to run against a local build process.
  2. Set up the npm metadata API to directly query and construct structured package metadata in the format used by the builder from npm.
  3. Set up a metadata persistence layer which can be customized to use various persistence layers, with a standard default option initially.
  4. Extract the JSPM CDN analysis pass into a custom function that runs locally to build up structured analysis data for any npm package, which is used as the third metadata input into the package build. Fully test this analysis infrastructure locally.
  5. Set up a new REST API on the JSPM CDN itself to be able to share its global npm analysis data with nodes. This global analysis data provides information about how packages are used, which in turn is used for optimization input based on knowing what the most common modules are in a package to use. This remains the only remaining centralized component, and in time should become redundant once the package.json “exports” field gains full adoption (that is, this service simply aims to automatically determine the subpath exports for packages that do not list them).
  6. Finally extract the JSPM CDN builder from the main codebase into the new codebase, linking in the three analysis data components together.
  7. Provide a local HTTP server interface for local testing and usage of the project.

Month 2 - Distributed Builder MVP Completion

  1. Get all builder tests to pass
    • Get all CDN tests to pass against the isolated builder project node locally. This involves refining all the steps of the previous month and ensuring everything is cohesively running together to pass all existing tests.
    • Solve any integration issues that arise.
  2. Refine the options for customizing the long-term builder project daemon behaviour. This includes options to configure:
    • Npm sharding / rules for which packages to build or not to build.
    • Whether to serve a local server or not
    • Whether to subscribe to the npm package feed to pre-emptively cache (push-based builds) or to build on-demand as requests are made to the server (pull-based builds)
  3. Adding custom persistence layers to the package build cache itself. Initially based on a local FS cache only.
  4. Major milestone: Complete the Distributed Package Builder MVP for the JSPM Project.

Month 3 - IPFS Workflow Bootstrapping

  1. With the distributed builder architecture fully completed, IPFS can now simply be implemented as a custom persistence layer for package metadata and package builds. The previous phases will have ensured this is a well-defined integration by this point of the project.
  2. Along with this work, support can be added to the JSPM Generator project to configure IPFS layers for providers. This will require some careful architecting to fit into the model, the exact planning of which needs to be taken into account.
  3. Major milestone: Local Builder Daemon alongside Import Map Generation with full end-to-end package execution testing

Month 4 - Complete & Polish Workflow Demonstrations

  1. Feedback from the various workflows to be taken into account and any adjustments, additions, changes or fixes to be implemented.
  2. Major milestone: Fully implement the three major end-to-end workflows:
    • A normal browser running import maps against an IPFS pinning service
    • Node.js Import Maps on IPFS via a custom loader
    • A custom browser build that supports IPFS directly
  3. Stretch goal - consider workflows for Deno or edge providers
  4. Stretch goal - Integrate the distributed builder into the import map generator project itself, so that the process of generating import maps could also self-host them on IPFS, acting as both a build verification process and decentralized node at the same time.

Month 5 - Project Launch

  1. Major Milestone: Complete the Distributed Builder Integration
    Fully integrate these new workflows into the JSPM project and core technical documentation, including:
    • API documentation
    • Workflow guides
    • Integrating the distributed builder and IPFS concepts into the core descriptions of the project throughout the website.
  2. Preparation of video demonstrations / possible conference presentations
  3. Blog post & launch announcement

Exact timeframes will vary by milestone.

Total Budget Requested

The JSPM CDN currently builds every single package on npm, which costs over $2000 worth of Google Cloud compute each month. We have sponsorships from 37 Signals, TeleportHQ, Socket.io, Framer, and Scrimba, which just covers exactly this amount, but this cost is gradually increasing while there are also remaining overheads of the foundation such that there is still negative cash flow of the region of -$1000 per month.

A portion of the grant ($10,000) would go towards sponsoring the JSPM Foundation directly. Guy Bedford does not take any salary or compensation from the JSPM Foundation, and will not be taking any fee for project management and the running of regular meetings with Guy Paterson-Jones so this would entirely go towards infrastructure costs for the JSPM project.

$30,000 / $5000 per month for 5 months.

Maintenance and Upgrade Plans

Since this project integrates IPFS into the JSPM core product, the JSPM CDN builder, maintenance will be guaranteed by the JSPM Foundation which is set up to ensure the sustainable maintenance and support of the entire JSPM project for the long-term. The project has been going for 10 years already, so there is some precedent for long-term support.

The project will not guarantee the hosting of an IPFS mirror of the JSPM CDN permanently on the JSPM infrastructure however, since this would add unnecessary costs to the already large infrastructure costs of the project. But the IPFS integration in the CDN node architecture as an option would continue to be maintained over time with the goal to switch over to it eventually in future as a long-term goal of the project to fully migrate to this approach.

Team

Team Members

Project-specific members:

The JSPM Foundation itself consists of the following members:

Team Member LinkedIn Profiles

Team Website

https://jspm.org

Relevant Experience

Guy Bedford has been involved closely with the growth of JavaScript modules in browsers. As a TC39 member he wrote the specification for top-level await and as a Node.js core contributor provided a significant amount of the implementation of Node.js’s integration of JavaScript modules, including implementing a lot of the Node.js module resolver code for ES modules. He has also been a long-time contributor to the import maps specification and is also the author of the import maps polyfill in browsers, ES Module Shims. Previously he created the SystemJS module loader for browsers. He is the creator of the JSPM project, the longest running dedicated native modules CDN for JavaScript, and the first to provide a CDN for import maps.

Guy Patterson-Jones collaborated with Guy Bedford on JSPM back in 2015, and while Patterson-Jones can usually be found climbing the mountains of Switzerland or thinking about Mathematical logic has been looking for a new project to get his teeth into recently. His recent experience working on the Bacalhau compute project make him uniquely suited in having experience in both the JSPM project and the Filecoin and IPFS projects.

Team code repositories

JSPM Projects

Guy Bedford Open Source

Guy Patterson-Jones Open Source

Additional Information

I’ve been a long-time follower of IPFS, and this has been on my mind for a very long time.

Contact guybedford@gmail.com to discuss the grant agreement and next steps.

I’m happy to arrange a meeting anytime to discuss the project details further. And questions / feedback / refinements to the proposal are also very welcome.

guybedford commented 1 year ago

The window for the viability of this specific project collaboration has now closed. Further work in this space is still very much on the roadmap, albeit longer term without there being a clear funding model in place. I'm always happy to discuss this space, if and when there is any interest.

ErinOCon commented 1 year ago

Hi @guybedford, thank you for sending an update regarding the project window. Do you have an interest in submitting a revised proposal that aligns with your current roadmap? If so, I would be very happy to include this in our next review cycle this month (as opposed to it being added to our queue).

guybedford commented 1 year ago

Hi @ErinOCon thanks for the suggestion here. The proposal would need to be reworked quite a bit for there no longer being this collaboration in place, but an alternative simpler project might be possible. Can you let me know what the deadline would be for submitting this new proposal? Would this repo still be the right place for that?

ErinOCon commented 1 year ago

Hi @guybedford, the next review cycle is expected to begin in the next couple weeks, but we would be happy to work with you if more time is needed.

If the updated submission is geared more towards a new project instead of a revised project, I recommend submitting a new proposal and sending notice on this thread when it is ready for review.

Happy to discuss further as needed!