feat: third-party dependency management through npm

[HipsterBrown]

Following through on the remote dependency "docs" and existing manifest includes management for Moddable modules, this is a proposal for managing third-party modules using npm and associated package.json as the standard shared by the JS ecosystem.

There are various levels at which this could work.

1. The npm CLI would be used directly to install dependencies for a project and xs-dev would handle configuring the manifest.json includes to point to the modules' manifest:

npm install dtex/j5e // adds j5e to package.json, pulled from GitHub
xs-dev include j5e/lib/led // adds the path to the j5e led dependency under node_modules

This takes advantage of the npm feature for installing modules from git, forgoing the need for module authors to publish their code to the central npm registry.

2. xs-dev provides a command to install dependencies using npm under the hood (see the gluegun packageManager module), and sets up an alias for the path to that module (see docs for the build field in the manifest):

xs-dev install dtex/j5e // adds j5e to package.json, creates J5E env variable for the path under node_modules under the "build" field in the manifest
xs-dev include j5e/lib/led // adds the path to the j5e led dependency from the J5E env variable

It is expected for folks using xs-dev to have npm (or some Node package manager) installed since that is the easiest way to get xs-dev at the moment. It is nice to collect the package management behavior under the single tool, just as xs-dev provides a command over CLIs like mcconfig. This is simply automating what could be done by hand, if needed. The gluegun feature will also detect yarn usage if that is preferred by the environment as well.

When working on a project with third-party modules, the xs-dev install command without any arguments will act the same as npm install and fetch any missing dependencies.

[dashcraft]

Something i've been doing more of recently is using the git import syntax for npm, where it links the github repository. It can be configured to match against tags and branches for instance, works decently well.

[HipsterBrown]

Inspired by the recent npm support added to Deno, rather than using npm as the standard for third-party package management, import maps can be supported by xs-dev and Moddable with namespaces for the package source, i.e. npm, git, etc. 🤔

[HipsterBrown]

Capturing some ideas from the recent TC53 meeting where we discussed the proposal for standardizing package.json and how we might introduce support through xs-dev first before building it into the Moddable tooling if it makes sense.

• During the build / run commands if xs-dev detects a package.json file, it will generate an associated manifest_package.json (or some such name) to be included in the main manifest.json.
• The manifest_package.json will resolve the dependencies defined in the package.json using a process similar to how Tessel would created "slim" bundles before deploying to devices via t2-project, which uses the resolve package under the hood.
• The separate file should prevent xs-dev from clobbering the configuration stored in the main manifest.json and avoid errors from users modifying the generated mapping unexpectedly.
• The main manifest.json can be used to configure the host environment, while the package.json is seen as the source of truth for the application configuration.

Some questions to answer during this prototyping:

• What happens when we find a manifest.json within a dependency?
• Do we enforce a "global-style" config with a .npmrc, otherwise how do we dedupe common dependencies among direct dependencies?
• How do we warn about binary / native dependencies that won't work with XS?
• Can we support a custom xs runtime key to resolve the correct entrypoint for the host environment?

cc: @dtex @phoddie

[phoddie]

@HipsterBrown – thanks for capturing these notes. I support the concept but the details are pretty hazy for me still. What do you think about a real-time working session with you, @dtex, me, and anyone else with an opinion (@dashcraft, @tve, etc) to work through some examples of what this might look like in practice?

[HipsterBrown]

I'm game for a "mob programming" session to get a proof of concept together. I've never used the video call functionality on Gitter but that might be a nice medium.

[tve]

If I may ask, I have some questions... I'm not super knowledgable about npm and its inner workings, so I may be completely off-base. Going back to the beginning, what are the goals? I see npm doing a bunch of things:

• there's a package.json format to specify dependencies (and much more)
• there's a repo with an API to store & retrieve packages
• there's a local filesystem tree with a specific structure to store fetched packages (and the ability to link locally developed packages)
• there's a dependency resolution mechanism that resolves multiple version specifications for the same package
• there's a remote fetch machinery that can retrieve missing packages or fetch updated versions of packages
• there are several bundling features to transform the file system tree into "exportable" formats

The Moddable SDK already has the ability to fetch "packages" from git, is the focus here on:

• a more powerful mechanism?
• some tooling around this, e.g. to update versions or perform dependency resolution?
• the ability to consume (non-git) npm packages having a package.json instead of a manifest?
• the ability to fetch packages from the npm repository instead of using the git protocol?
• something else?

[phoddie]

I'm game for a "mob programming" session to get a proof of concept together. I've never used the video call functionality on Gitter but that might be a nice medium.

Cool. I'm flexible on time and venue. I've never used video on Gitter either. I added a video conference link to the channel. It seems to use Jitsi which should work. I'll leave it enabled for a while. If you (or anyone else) wants to connect we could quickly see if that works.

Code would be great, but even putting together some examples of meaningful / useful package.json as a target for implementing would be a valuable output. That is probably obvious to you. It would also address some of the questions @tve raises, which are largely about scope.

@tve, from my (perhaps naive) perspective the goal is primarily about providing a convenient way for developers already familiar with package.json to apply that knowledge to their Embedded JavaScript work. It may also bring some new capabilities to the Moddable SDK, such as the ability to use compatible npm packages directly. My sense at the moment is that this could be really nice for building projects. It isn't likely to supplant the manifest which has more than a few features which are outside the scope of package.json (as I understand it) but essential for building an optimized embedded firmware.

[HipsterBrown]

I appreciate the questions @tve 👍

One of the core advantages of integrating with npm is familiarity for the folks in the JS ecosystem. The registry is the de facto repository for sharing packages, so there is a lot of tooling for publishing, consuming, scanning, etc that doesn't come from a git source (although package.json supports that as well) and wouldn't have to be managed by Moddable tooling. Efficient caching, deduping, and conflict resolution can be handled by standard tooling.

I believe packages can have a package.json and manifest.json together, since they can serve different responsibilities as described in one of my above comments. One comparable system is web extensions, which can use a package.json to describe the dependencies used by the project at build time, along with a manifest.json to configure the capabilities and files for the deployed package. https://developer.chrome.com/docs/extensions/mv3/intro/mv3-overview/

[tve]

WRT "a convenient way for developers already familiar with package.json to apply that knowledge to their Embedded JavaScript work": what this brings to mind is that npm provides a lot of features and other tools integrate with it. For example, the config for npm run or integration with testing frameworks for which one sometimes adds stuff into package.json. But all of this really pretty much presumes that one can run npm and not that there's a separate tool that replaces npm. So them new tooling would focus on pre-processing the package.json into manifests for the Moddable SDK?

[phoddie]

So them new tooling would focus on pre-processing the package.json into manifests for the Moddable SDK?

More-or-less, yes. (I would give a firm "yes" but I want to leave room to explore.)

[HipsterBrown]

But all of this really pretty much presumes that one can run npm and not that there's a separate tool that replaces npm. So them new tooling would focus on pre-processing the package.json into manifests for the Moddable SDK?

Correct. There are multiple interfaces for consuming a package.json, including npm, pnpm, yarn, various node modules for programmatic access and controls. xs-dev could provide some extra validation and discovery on top of npm but ultimately be responsible for generating a consumable manifest for mcconfig. This aligns with one of the core tenets around xs-dev: it should be possible to do without, but xs-dev makes it easier and seamless. If any of the xs-dev commands don't function how you want, you can drop down to the core tooling (mcconfig, mcrun, etc) without hassle.

[phoddie]

@HipsterBrown – One of the questions that I don't have clarity around yet is where the generation of a manifest for mcconfig would occur. As @tve notes, mcconfig today can already pull in a Git repository. Isn't much of a stretch to pull in an npm package. And mcconfig today generates a manifest for Node-RED projects from that project's flows.json (which, ironically, indirectly references npm packages). I'm entirely flexible on where it gets done and how that's implemented, but do think there's some exploring to do to sort that out.

[HipsterBrown]

@phoddie my original thought was to prove out the concept and requirements at the xs-dev layer first, then provide a more complete proposal for the mcconfig integration, if reasonable. This is the same thought around the platform.json idea that's been sitting in a draft PR for a bit.

The prior art related to node-red is good context to set expectations for a package.json integration.

We could try using j5e and that recent node-cron port you developed as example packages for the pairing session. I should have some time next week to dedicate towards that endeavor.

[phoddie]

@HipsterBrown – that's a reasonable approach. It is probably a bit easier to at least start in xs-dev.

I should have some time next week to dedicate towards that endeavor

Should we schedule a tentative time to talk / work / review?

Separately - you had asked about contributing my node-cron port back to the origin. The major impediment there, that I see, is that Cron is implemented with CommonJS. It was trivial to switch to standard ECMAScript modules, but I suppose the distribution will stick with CommonJS for compatibility. Cron depends on Luxon which would also needs some changes. Those are small by comparison. But without both, it is a problem.

FWIW – this challenge is also related to our Node-RED efforts. There are nodes which won't run on a microcontroller as-is, but it is pretty easy to create a version that will (Open Weather Map is a good example - I reworked it to use fetch instead of the original Node.js http. The change is straightforward but upstreaming is unlikely. In that case, the code in mcconfig remaps the node to use the MCU version of the module so the substitution is invisible to developers. This all suggests to me that perhaps there's a need for a mechanism that can map from an npm package to an Embedded JavaScript compatible version of that same package. (Lots of handwaving goes here!) For packages that can be update, it won't be needed. But it would "just work" for developers and create a low-friction way of delivering MCU compatible implementations of common modules. You may be horrified by the idea and/or you may be aware of precedents that do something similar.

[HipsterBrown]

@phoddie

Should we schedule a tentative time to talk / work / review?

Sure thing. Do Gitter support polls? 😄

The major impediment there, that I see, is that Cron is implemented with CommonJS. It was trivial to switch to standard ECMAScript modules, but I suppose the distribution will stick with CommonJS for compatibility. Cron depends on Luxon which would also needs some changes. Those are small by comparison. But without both, it is a problem.

The CommonJS legacy will be the biggest hurdle for folks wanting to use popular packages from npm, as there is still a lot of work for maintainers to support ESM by default. This will need to be communicated very clearly as part of this feature and in any error messaging.

This all suggests to me that perhaps there's a need for a mechanism that can map from an npm package to an Embedded JavaScript compatible version of that same package.

This is pretty much what the exports field in the package.json aims to solve within a single package. As for packages that can't / won't be updated with a embedded-compatible export, there could be a convention for finding an embedded equivalent in the same manner as the DefinitelyTyped project for sharing community-authored type definitions for related packages. npm will actually indicate if a package has a associated "DT" equivalent to be installed alongside it. I'm not suggesting a massive monorepo, rather a documented naming convention for discovering a community-authored package like node-cron-xs or node-cron-embedded. Prior art can be seen in the Babel and ESLint ecosystem of conventional plugin package names.

[phoddie]

Do Gitter support polls? 😄 In the fine Ecma tradition. I set-up a Doodle. Let me know if any changes are needed.

The CommonJS legacy will be the biggest hurdle for folks wanting to use popular packages from npm....

Yea... this was a big enough issue with Node-RED that I did a very simple, very terrible hack that lets basic CommonJS Node-RED node implementation work. But.... really.... standard modules are the way to go.

This is pretty much what the exports field in the package.json aims to solve within a single package

I'll have to learn about that.

As for packages that can't / won't be updated with a embedded-compatible export, there could be a convention for finding an embedded equivalent in the same manner as the DefinitelyTyped project for sharing community-authored type definitions for related packages. npm will actually indicate if a package has a associated "DT" equivalent to be installed alongside it. I'm not suggesting a massive monorepo, rather a documented naming convention for discovering a community-authored package like node-cron-xs or node-cron-embedded. Prior art can be seen in the Babel and ESLint ecosystem of conventional plugin package names

Good models, thanks. I do think discovery is an issue, separate or otherwise. As examples, there's no easy way for developers to discover my Cron port or @tve's spiffy Fusion port and BMI160 IMU ECMA-419 driver or the SD Card integration @salarizadi did for ESP32.

[HipsterBrown]

I do think discovery is an issue, separate or otherwise

Looking at other projects for reference, DeviceScript provides a guide for package discovery: https://microsoft.github.io/devicescript/developer/packages#npm-packages

They also include a command line helper for automating the preparation of a project as a discoverable package to work with their ecosystem: https://microsoft.github.io/devicescript/developer/packages/custom

I believe both of these steps are possible for Moddable and xs-dev together. There may be a couple levels to think about in terms of conventions and compatibility:

• works with Moddable / XS
• works with ECMA-419 compliant runtime

Focusing on xs-compat is probably the way to go at first, until more runtimes and agnostic packages are available.

[markwharton]

Another perspective...

It's several years old now, but KinomaJS supports Node.js module resolution with a KPR_NODE_MODULES build flag.

https://github.com/Kinoma/kinomajs/blob/701879d37e7fe5001420e0053cd60df6b91e4553/kinoma/kpr/kpr.c#L186-L193

https://github.com/Kinoma/kinomajs/blob/701879d37e7fe5001420e0053cd60df6b91e4553/kinoma/kpr/kpr.c#L251-L457

It's based on Node.js v4.2.1 require.resolve() pseudocode and enables KinomaJS to import Node.js modules.

https://nodejs.org/dist/v4.2.1/docs/api/modules.html#modules_all_together

So yea, like I said it's old. I just checked the latest (now Node.js v20.3.1) and it includes LOAD_INDEX, LOAD_PACKAGE_IMPORTS, LOAD_PACKAGE_EXPORTS, LOAD_PACKAGE_SELF, RESOLVE_ESM_MATCH.

https://nodejs.org/api/modules.html#modules_all_together

I know this discussion is about xs-dev, but would it help to resolve Node.js modules natively in Moddable SDK?

[phoddie]

Some of us at Moddable have been thinking about this a bit more.

To state the obvious: the road to achieving these goals inevitably requires passing through the process of resolving Node module specifiers. As @markwharton notes, there are a very well defined algorithms for resolving Node modules. That was once possible to support in XS, as the Kinoma links show. We just completed an experiment that confirms it is still possible with the current algorithms.

Following @HipsterBrown's idea of a separate tool, we built a small front-end that runs before mcconfig or mcrun and follows the imports and exports from the package.json to create a Moddable manifest. One important function of the manifest is as a module map.

As @HipsterBrown also outlined, the runtime keys of the package.json are used. They can select between the CommonJS and ECMAScript module versions (XS always chooses ESM) and allows for an "xs" or "moddable" runtime key.

Most of the work was in the tool, specifically on module resolution. Doing that correctly requires parsing the JavaScript source code. Since XS has a JavaScript parser, we used that (you could do the same with any number of Node modules).There are also some small supporting changes to the Moddable SDK, mostly isolated to mcmanifest derived tools and the module resolution algorithm in the hosts (as you might expect).

Effectively what we have is a bundler. Unlike a web bundler -- which bundles everything together for efficient use in source code form on the web's JavaScript runtime -- it bundles everything together for efficient use in binary form on Moddable's JavaScript runtime.

The tool itself isn't ready to share. We are trying to make time to get it there. I thought I'd share this information now to make sure everyone in this discussion is aware of the effort and its current progress. And, of course, to hear your feedback and questions.