search

uNetworking / uWebSockets.js

μWebSockets for Node.js back-ends :metal:
Apache License 2.0
8.13k stars 581 forks source link

Selective binary by OS and Node version 🤔 #1112

Open rtritto opened 1 month ago

rtritto commented 1 month ago

After uWebSockets.js is installed, all OS and Node versions are built.

Reproduction

Expected

In directory /.yarn/unplugged/uWebSockets.js-https-<HASH>/node_modules/uWebSockets.js there is only 1 file filtered by OS and Node version

uNetworkingAB commented 1 month ago

It downloads all binaries because npm client has no idea of the platform, no support to download only a subset of binaries like literally all other package managers on earth do. So it's a tradeoff between being easily downloaded with npm client and saving disk space.

rtritto commented 1 month ago

So the issue is both npm and yarn. I opened an issue to Yarn Berry https://github.com/yarnpkg/berry/issues/6565

rtritto commented 1 month ago

@uNetworkingAB

https://github.com/yarnpkg/berry/issues/6565#issuecomment-2426429147

It's not an issue in Yarn Berry or any other package manager. The issue is how uNetworking/uWebSockets.js is distributed. They don't use os and cpu fields of package.json and instead distribute all platform specific files in a single package: https://github.com/uNetworking/uWebSockets.js/tree/v20.49.0

larixer commented 1 month ago

It downloads all binaries because npm client has no idea of the platform, no support to download only a subset of binaries like literally all other package managers on earth do.

The platform can be communicated to npm client via os and cpu fields in package.json. The JavaScript package managers should also support installing packages from GitHub subfolders. Combining these two features might work in your case to decrease network bandwidth during package installation at a price of some complication of releasing process on GitHub. E.g. you can keep binaries in some branch, but use multiple subfolders, like native-linux-x64, native-darwin-x64, etc, each subfolder having package.json with os and cpu fields for the target platform. Platform-agnostic and user-facing package can be kept in some other branch and should be tagged on each release as you do it now. User-facing package can then have all binary packages corresponding to its version as dependencies or optionalDependencies. Will this work well with all JavaScript package managers or not is hard to tell, in theory - it should, I am not aware of the package that do it already, so, YMMV and it's an open question if additional complexity for distribution is worth it.

uNetworkingAB commented 1 month ago

Very interesting! Do you think it will work with the same user experience?

npm install uNetworking/uWebSockets.js#v20.48.0

and

require("uWebSockets.js")

?

larixer commented 1 month ago

@uNetworkingAB Yes, it should work with the same user user experience.

porsager commented 1 month ago

It will still download the various abi versions, but that's still an improvement! I can make a quick sample setup for you to see

porsager commented 1 month ago

In my projects I'm just using this loader and a postinstall to only download a single binary, so that's an option too:

import fs from 'node:fs'
import path from 'node:path'
import https from 'node:https'
import { pipeline } from 'node:stream/promises'
import { createRequire } from 'node:module'

const binaryName = `uws_${ process.platform }_${ process.arch }_${ process.versions.modules }.node`
const binaryPath = path.join(import.meta.dirname, binaryName)

fs.existsSync(binaryPath) || await download()

let uws
try {
  uws = createRequire(import.meta.url)(binaryPath)
} catch(e) {
  await download()
  uws = createRequire(import.meta.url)(binaryPath)
}

export default uws

async function download(url = 'https://raw.githubusercontent.com/uNetworking/uWebSockets.js/v20.47.0/' + binaryName, retries = 0) {
  return new Promise((resolve, reject) => {
    https.get(url, async res => {
      if (retries > 10)
        return reject(new Error('Could not download uWebSockets binary - too many redirects - latest: ' + res.headers.location))

      if (res.statusCode === 302)
        return (res.destroy(), resolve(download(res.headers.location, retries + 1)))

      if (res.statusCode !== 200)
        return reject(new Error('Could not download uWebSockets binary - error code: ' + res.statusCode + ' - ' + url))

      pipeline(res, fs.createWriteStream(binaryPath)).then(resolve, reject)
    })
    .on('error', reject)
    .end()
  })
}

If the postinstall doesn't run, it'll download it on first run after instead.

larixer commented 1 month ago

@porsager @uNetworkingAB I have put together a demo repo: https://github.com/larixer/multi-platform

You can check how it works via: npm init npm add larixer/multi-platform#1.0.0

You can then go into node_modules and see that user-facing multi-platform and one for your current platform package is installed, provided, if you are on Linux, Win32 or Mac.

Note, that I have used 4 branches, 1 for user-facing package main, and 3 other for linux-x64, darwin-x64 and win32-x64, because in fact support for Git subfolders is not that great in JS package managers, but they support tags and branches and this should be enough.

rtritto commented 1 month ago

There is a way to select also by Node version?

larixer commented 1 month ago

There is a way to select also by Node version?

There is engines field where you can put required Node version, but as far as I know npm will not skip installing the dependency if Node version is not compatible, it just warns about Node version mismatch, so it's not possible to use engines like os and cpu to selectively skip dependency installation.

rtritto commented 1 month ago

There is a way to select also by Node version?

Can we use the release tag?

Eg:

uNetworkingAB commented 1 month ago

You can have multiple nodejs versions on the same platform so installing all 3 is preferrable IMO.

Thanks for making the demo, it works for me on Linux but I had issues getting it to work on arm64 macOS. I made a fork but could not get it to work for some reason. Anyways -

If someone wants to make a full demo, using the actual binaries in latest release, and making it so that require("uWebSockets.js") works on macOS arm64, x64, Windows x64, Linux x64, arm64 - please do so and provide a branch I can test against. If that branch works, I will definitely move to this approach, as it would be a seamless optimization.

My point re. full demo is that I do not have time to play with this and this is a very low prio issue. So I can take a look at the finished full demo if someone makes it.

webcarrot commented 1 month ago

Off topic: Binary without SSL and H3 (pure http 1.1) would also be nice.

uNetworkingAB commented 1 month ago

That's way too specific. If you value disk space, use ws.

porsager commented 1 month ago

Ok - I've made 2 PRs ready master branch changes and binary branch changes, and a sample release on my fork you can try out.

npm i porsager/uWebSockets.js#v20.50.0

I've tried to keep it as close to the current setup as possible.

I've left the current build.yml script completely as it is, and added a new release script that allows you to release by running it from the actions tab and inputting the version you want to release. It takes care of making git tags and individual package.jsons for every single binary, and the main package.json with optionalDependencies that makes package managers only install the needed binary.

https://github.com/user-attachments/assets/8e4b01fb-29f4-4bbe-b348-56f2e1e881d4

If anything fails you can simply delete the tags, and run again.

Only downside is all the friggin tags, but I think that's definitely worth the tradeoff.

The only maintenance should be in adding new ABI versions to the version map in release.yml when a new node version should be supported.