feat: allow package manager selection upon install

[sandstrom]

Summary

Allow selection of package manager (npm, yarn) during setup.

Motivation

NPM, with all its flaws, is still the most popular package manager. Also, the package managers come and go. Having a standardized command for selecting one would make it easier to add more in the future.

Guide-level explanation

• Remove or deprecate --skip-yarn and --no-skip-yarn flags.
• Add a --js-package-manager flag, which would accept js-package-manager={npm,yarn,none}. Selecting 'none' would be the same as the old --skip-yarn flag.
• The default behavior, when the flag is omitted, could still be to select yarn. It's not the default I would have chosen, but that's a minor thing, as long as changing is easy.

Reference-level explanation

Could add more details if this suggestion receives positive feedback.

Drawbacks

It's possible to remove yarn and install npm manually.

Unresolved Questions

n/a

[jaredcwhite]

Thanks for submitting this issue!

We're only ever going to support one JS package manager. There are many hooks in the system which expect a certain CLI to be available, so if it's not Yarn it would be something else exclusively. And a long-term goal of the project is to find a way to abstract this entirely and not rely on a system-installed package manger at all (aka switch to Ruby tooling for all high-level interfaces).

I appreciate the idea, and we'll keep an eye out for how this space continues to develop.

[sandstrom]

Alright, if you're only going to pick one, I'd totally go with NPM. I know NPM has many flaws, but it's still the de-facto Node package manager. Also, with Github/Microsoft now running the NPM project, it's likely to remain well supported and stable for the foreseeable future.

Any default choice should be the one with the least friction. Everyone knows NPM and how it works (even though they may not have it as their preferred node manager). But not everyone knows how Yarn or pnpm works, so that's always going to be more of a threshold for newcomers.

---

As an aside, I understand the desire to abstract it away, but my guess would be that it's a lot of hassle. Better to not abstract too much over the front-end stuff, instead threat them as two separate processes, that produces different output (build artifacts).

---

Great work on BridgeTown, an awesome project! 🏅 We're doing some tests now, to see if we can migrate our Middleman site over to Bridgetown.

[sandstrom]

Another argument/anecdote for not abstracting over Yarn/NPM too much.

The Cordova project (build mobile apps with JS) spent tons of effort trying to abstract over iOS and Android build pipelines. It was a constant hassle, with a lot of work. Eventually the sibling/fork Capacitor started, which basically gave up and used 95% native build code from each project. It has worked much better!

I think it's better to treat it as a peer dependency or programming interface. Basically, that Bridgetown expects that the front end dependency manager will produce a populated node_modules directory when the command defined by js-package-manager-run is called. It will exit with code 0 if successful. And so forth.

That makes it much easier to customize, and puts less of a burden on Bridgetown.

[jaredcwhite]

@sandstrom I'm starting to come around to your way of thinking. 😄 If we can come up with an abstraction—some kind of adapter pattern or whatever—that lets us keep our current usage of Yarn but makes it easy to use just NPM or something else in the future, I'm down with that. I wouldn't necessarily want to maintain all those adapters, but maybe it's something the community as a whole can own.

Do you have any thoughts of how Bridgetown's build process could know which package manager is in use for a project after the initial new command? Do we make that a config option? Add some sort of dot file somewhere? Detect via yarn.lock vs package.lock vs whatever? (seems the most brittle)

[sandstrom]

Yes, I think when it comes to front-end stuff, Bridgetown should certainly strive to work together with the popular tools. But doesn't necessarily need complete integrations where Bridgetown knows about all available tools, and how to invoke them individually.

In general, with large open-source projects and integrations, it's typically much less work to provide the primitives for the integration between X and Y to be possible, and then implement the integration via docs/guides based on those primitives.

For example, maybe it's common for most front-end tools to run a build step as part of the overall bridgetown build command. Instead of having ruby-code invoking packer, snowpack, or other binaries directly, based on a configuration option choosing among a list of predefined front-end tools, maybe we can get away with using a ruby hook + a configuration option? E.g. frontend_build_command: 'yarn install' where the default value can be replaced with snowpack install --compress or whatever.

We'd provide the primitive in the form of a hook and the config option frontend_build_command. Then, in the website docs for Bridgetown, you can have different sections for different front-end tools, explaining how to integrate them and the steps to take.

Such an overall approach will be much easier to maintain over time. It'll also be more flexible, because if a new glacierpack frontend tool emerges, they'll probably just have to update the config option. They won't need to PR a change to the ruby code and verbatim add that particular command.

---

Good question!

Maybe a new command with a flag such as --js-package-manager=npm could also add js_package_manager_install: 'npm install' as a entry into the bridgetown.config.yml that the new command will create.

Even better (but don't know if that's possible) we could simply say:

Bridgetown expects a directory node_modules to exist in its root. You can use whatever package manager you want to populate that directory. In our docs you can see examples of how to use NPM and Yarn.

And then check if that directory exists when starting the front-end stuff. If that directory doesn't exist, show a warning message. That way, Bridgetown wouldn't really be concerned with how that directory is populated. pNPM, NPM, Yarn, Lerna, etc. -- anything goes.

[jaredcwhite]

@sandstrom I'll do a review of where we currently assume a Yarn environment and add a list here of places we'd need to update. Hopefully it's less of an issue than looms in my mind at present. I agree this would ultimately be a win. For the most part Bridgetown really doesn't care what's in node_modules at all…all it cares is if it can kick off a script to run esbuild or webpack.

[halostatue]

@jaredcwhite I’m currently working on something for which I might adopt Bridgetown for the documentation bits, as it’s a multi-language monorepo (three reference implementations of a spec). From my perspective, the biggest downside is the use of yarn, as I’ve adopted pnpm for package management.

That said, there’s very little work that would need to be done to be compatible with both Yarn and PNPM:

1. The tests probably don’t need changing, unless they’re checking yarn.lock; the lock file format for each item is different, and that parsing (if implemented) would need to be changed. All three package managers support <bin> list --json to get a JSON representation of all dependencies installed. There are differences in the output—the weirdest is Yarn's. I can’t quite parse what the test in bridgetown-core/test/test_plugin_manager.rb is doing.
2. There should be a Bridgetown.js_package_manager function (or something) that returns the configured package manager (yarn, npm, or pnpm). This would be used in things like bridgetown-core/test/test_esbuild_command.rb:113 instead of matching against just yarn run esbuild-dev.
3. Output references to Yarn like that on bridgetown-core/script/update-site-template:9 should be rephrased to either Node dependencies or JavaScript dependencies.
4. The js_package_manager function would also be used in the generated Rakefile. Instead of sh "yarn run esbuild", it would be something like sh "#{Bridgetown.js_package_manager} run esbuild".
5. PNPM and Yarn use <bin> add. Modern versions of NPM have npm add as an alias for npm install, so using `#{Bridgetown.js_package_manager} add -D #{plugins.join(' ')}" would be sufficient.
6. bridgetown-core/lib/bridgetown-core/configurations/shoelace.rb has bare run-script commands. This is legal in PNPM and Yarn, but not legal for Node. Changing gsub_file "package.json", %r{"esbuild-dev": "node}, '"esbuild-dev": "yarn shoelace:copy-assets && node' to gsub_file "package.json", %r{"esbuild-dev": "node}, '"esbuild-dev": "#{Bridgetown.js_package_manager} run shoelace:copy-assets && node' would be sufficient to make this work.
7. Replacements for --skip-yarn-install would be needed, as well as a configuration option to configure and override the package manager. Defaulting to yarn is still a reasonable default, IMO.
8. yarn remove is a Yarn-ism, but both NPM and PNPM support it as an alias for uninstall.
9. When executing a binary (not a script) installed with a package, <package-manager> exec <binary> is safer. PNPM and Yarn do the smart thing (check for a script, then check for a binary), but NPM b0rks if you want to exec something but instead run it.

There appear to be only about 50 files to edit, and some of those are blog posts, so can be ignored.

The only other issue appears to be that, somehow, yarn is intuiting the need for a development package read-cache, which pnpm is not (probably because pnpm behaves somewhat more strictly when it comes to transitive dependencies). Doing pnpm add -D read-cache lets pnpm run esbuild work.

[jaredcwhite]

@halostatue Wow, this is a great rundown and essentially can guide us where to go next.

I'm presently swamped with an overhaul of the Ruby side of things re: initialization & configuration, so I would gladly welcome a PR from anyone with at least some steps taken to further this along. Otherwise it'll be a bit of time before I could start to dig in.

cc @sandstrom

[sandstrom]

@halostatue Great summary and overview of JS dependency integration points!

The path I'd pursue (and please argue against me if you don't agree, I may have missed something) is that Bridgetown should try to think about the JS layers mostly in terms of contracts and expectations.

The JS landscape is quite volatile. There are many package managers (esbuild/vite/rollup), competing JS runtimes (node/bun/deno) and competing dependency tools (npm/yarn/pnmp). This innovation is good, but Bridgetown should be clever and try to avoid picking sides, or wrapping any of them.

So for JS package managers, Bridgetown could say we have these N expectations (these are examples):

• The command configured in js_package_manager_install should return the 0 exit status if successful, and should populate the directory node_modules in root with JS dependencies.
  • For differences between environments, we also support js_package_manager_install_{development,test,production}.
• The command configured in js_build should return 0 and populate the directory output/js with…

These are just examples, but you get the idea.

All of these contracts doesn't have to be in code. For example, some of the current plugin logic, I'd replace with docs + CLI-instructions. Basically, instead of having some plugin run npm install commands by itself, it should simply emit a CLI output message telling the manager to install the package themselves.

For example, number 5 in the list above sounds like a ruby <> JS integration that we don't need to maintain in code.

5. PNPM and Yarn use add. Modern versions of NPM have npm add as an alias for npm install, so using `#{Bridgetown.js_package_manager} add -D #{plugins.join(' ')}" would be sufficient.

---

The catalog of integration points above is great. I think a good first step might be to catalog them (maybe that list is already exhaustive) and then discuss if any of them can be removed altogether (handled by docs/instructions/cli-output) and for the ones we need, how could we make that more of a contract than a direct integration. Basically, define what expectations we have on that JS component, and then make it easy to customize which command will be run.

[sandstrom]

@jaredcwhite Added us as a sponsor just now.

Hopefully this helps towards some of the ideas expressed above (https://github.com/bridgetownrb/bridgetown/issues/609#issuecomment-1221277335), assuming you think those ideas are good. As explained above, I think BT should try to surf along side JS frameworks, but don't wrap them or tie itself too closely to particular frameworks.

[jaredcwhite]

Much appreciated @sandstrom! 👏🏻 This will definitely be a top priority for Bridgetown 2.0 (or 1.3?) after we finish up our current release cycle.

[KonnorRogers]

Coming back this, 👍 , currently ive been hacking by using bin/bridgetown start --skip-frontend and having a Procfile run esbuild separately, running overmind nested in overmind is admittedly not great.

[jaredcwhite]

@KonnorRogers Hmm at this point I'm actually not aware of any issue preventing you from running pnpm directly from your Rakefile, so I'm not sure why you would need to skip Bridgetown's frontend process.

[KonnorRogers]

@jaredcwhite if you have plugins, it always tries to run the yarn install command. I can try again with the latest release and see if thats still the case

[jaredcwhite]

@KonnorRogers ah, I think that's a function of if a gem and whatever's in package.json aren't in agreement version-wise. Maybe we should look into making that particular feature optional in a point update.

[KonnorRogers]

@jaredcwhite I found some skip_yarn references in there, but can't quite figure out where it's getting referenced

[wilsonsilva]

What do you think about replacing Yarn with Bun? Bun is a lot faster, it is updated multiple times per week, has a stable API and does a lot out of the box.

[jaredcwhite]

@wilsonsilva I think the plan is to migrate away from Yarn in favor of the simplest/most universal install possible, which is simply Node/npm. @KonnorRogers has done some work to support pnpm as well which we we'll be releasing shortly. I don't have any experience with Bun personally and have no future plans to use it, so it wouldn't be something I can support. Appreciate the question though.