[Idea] Splitting forge

[alexandercerutti]

Hello everybody! I know, probably what I'm going to say here will sound weird to many, but I'm talking as a user and I don't know very deeply the internals of forge.

I'm here today to open a discussion that might be interesting IMHO. As almost every node.js developer knows, NPM packages are sometimes very big and creating a package that has one of them as a dependency, will bring them to weight a lot in the final project.

I use forge for my package passkit-generator. If we analyze it through dependencies analyzers like BundlePhobia and PackagePhobia, in case of a package of mine, which is not that big, we can see that, according to BundlePhobia, in my case, forge is responsible for the 39% of the space occupied if no other packages get installed. Through PackagePhobia, we can see that my package's install size is of 4,90 MB, even if itself is pretty small as code (I know, I have other dependencies, like moment.js, which seems to weigh a lot more). If we analyze forge through Packagephobia, it reports (as it has no dependencies) an install size of 1.61 MB, which will grow for sure (please note that that weight corresponds about to the size of the physical folder's weight, so with minified code) as other features must be created yet.

So, defined above, I'd like to ask this: why not splitting forge in multiple packages? I mean, to use a lodash-like approach (@forge/x509, @forge/pkcs as possible examples).

This would allow packages to bring way less weigh with them. Also, since they seem to be well divided, we can see that including for example pkcs (like, all of them), will include as dependencies like as they are imported inside the files themselves. Taking pkcs7.js's dependencies as examples, they might become:

@forge/aes;
@forge/asn1;
@forge/des;
@forge/oids;
...

Of course, all the files should be analyzed to know if they should be exported as single package or internal package file.

I know this might be difficult, but it might be a good way to improve this package in my opinion. Let me know what you think about it. Thank you!

[davidlehn]

We're aware of the browser bloat problem. I'm pretty sure everyone who uses forge on the browser side is aware! I've run into the problem a few times in recent days. Interesting analyzer sites, I'll have to keep those in mind.

You can do custom builds that may greatly reduce the size, but that's a hassle. That's mentioned in the README and there are some commented out examples in the webpack config. https://github.com/digitalbazaar/forge#building-a-custom-browser-bundle

The problem with many packages is that we'd like to keep the associated maintenance issues to a minimum. I'm afraid using lots of packages will take too much time to deal with. Not having tried that approach myself, I'm not sure what the issues would be.

I think what will happen near term is that we'll try some major refactoring and move everything to ES Modules and use the fine esm package for Node.js support. That would enable tree shaking tools like rollup and webpack to do their thing at a higher level. That might not cover everyone's use case, but the world is moving that direction, and in many cases would be a huge win for minimal library user effort. A conversion side effect will likely be the code will become easier to make into multiple packages if we went that direction later. It will also mean we can easily break out some of the cruft in the util file that I imagine no one in the world has ever used in production, including us authors.

[alexandercerutti]

Thank you for the prompt reply.

That is not a problem concerning browsers only, but also general Node.JS' apps (even if, I know, they run only on the server, so someone might think "who cares?"). Also, concerning the Browser, if someone uses Webpack, as per this issue topic, it should be able to include in the bundle only the imported ones.

While I was writing the issue topic, I was thinking about what you are saying. Yes, this might open to issues-handling problems, but everything should be organized before acting through this way. Anyway, you might still adopt an approach like "we have one repository, but we publish multiple packages", and that repository (so, the current one), is organized like a mono-repo (e.g. with lerna, if needed). Therefore, you'll be still able to keep all the issues in one place.

If you see, this is about the same approach that lodash uses. NPM is full of lodash.<package-name> packages (like, lodash.merge), but all the files are in one repo, in the main directory (or, following another git branch / tag, each module is handled as single folder - but I like most the mono-repo approach or anyway, an approach with a single folder containing all sub-repositories).

Please note that ESM module support for Node.JS is still experimental, and I don't know when it will be available. Also, of course, it won't be retro-compatible. But it should be anyway a good choice once it will be stable. Also, I think that splitting it into different packages, will make it easy to optionally convert it to typescript.

EDIT: I just read again your reply. With "maintenance issues" do you mean the ones you authors open, the ones that user opens, or both (as I was referring above)?

EDIT 2: I understood now that esm is a package to convert ESM syntax to supported ones. I didn't know it. Might be interesting...

[davidlehn]

The "maintenance issue" is spending any additional time at all on things like packaging. That's no fun. We'd rather spend limited resources on fixing bugs and adding features. I think any approach we do will start with a ES Modules conversion, or at least an attempt at it, so we can see what kind of improvement we get and re-evaluate if more needs to be done. Based on bug reports in the past there are people who still use this code directly in ES5 systems (ie, no let or const usage). Dealing with that alone after ES Module conversion could lead to packaging burnout. ;-)

[alexandercerutti]

Oh, I confused "issue" in "maintenance issue" with "issue topics" instead of "problems", sorry. I understand. What is "packaging burnout"? 😅

Anyway, I saw that the current package brings within, some minified files, the source-maps and also the files in lib/. Are they really needed? Aren't the minified files enough? I think, and correct me if I'm wrong, that removing lib/ would make it decrease drastically the library size.

[jimmywarting]

+1 for ESM cjs and extension-less require() is bad. it's not how the web works so it's hard for any client side http resolver to be able to figure out the real path of require('/foo'). is it: foo.js, foo/index.js or foo.json?

it's a anti pattern and a footgun that should never have been supported in the first place. even if you use cjs, you should actually be using explicit path, cuz the program can load faster then. and dose not have to 2nd quese what file needed to be imported by stat:ing and scanning all floders for what you need

[alexandercerutti]

I almost forgot about this issue I opened. Wow, almost four years have passed and the Javascript ecosystem has changed a lot!

Now using ESM modules might be really doable, also because they are actually tree-shakable (there were a lot of things I didn't know four years ago, so I cannot believe what I wrote in my previous texts lol). I am still of the idea that splitting packages would be really awesome. It would allow creating bundles with everything or feature-specific bundles or installing what is needed.

I might think of working on a POC just to show how this would end up.

[davidlehn]

I did a first-pass at ESM conversion long ago. Needs updates. It was rather brute force just to see it work. There are probably much better ways to rework everything. The very ancient forge API design does things that make clean ESM conversion a bit difficult. But hey, it did "work". The main maintainers of this lib (including me) have had other priorities and haven't completed that work.

https://github.com/digitalbazaar/forge/tree/es-modules

[PierreJeanjacquot]

Besides the ESM conversion which would be great, the main difficulty when it comes to picking the right parts is the side effects running in every modules.