ui dependencies in package-lock.json missing resolved/integrity fields

sarcasticadmin commented 3 months ago

Wave SDK Version, OS

Im attempting to package up waved and components from v1.1.1 tag. My system is using the following versions:

$ node --version
v18.19.1
$ npm --version
10.2.4
$ nixos-version
23.11.20240117.8bf65f1 (Tapir)

Actual behavior

serveral ui dependencies are missing integrity and resolved fields. Some examples https://github.com/h2oai/wave/blob/5fd6aa5447859e9266ed42a6f9b0583281e62c63/ui/package-lock.json#L17944-L17946 and https://github.com/h2oai/wave/blob/5fd6aa5447859e9266ed42a6f9b0583281e62c63/ui/package-lock.json#L17994-L17997

These missing fields causes the nix build process to fail since it prefetchs and verifies each of the dependencies then performs a npm build offline in a sandbox to ensure reproducibility. This is the result of nix-build against v1.1.1 for the ui:

$ nix-build
this derivation will be built:
  /nix/store/0hj7a05h8kvj8h5zhmac393242nn4pf6-waved-ui.drv
building '/nix/store/0hj7a05h8kvj8h5zhmac393242nn4pf6-waved-ui.drv'...
Running phase: unpackPhase
unpacking source archive /nix/store/fm6brwgw7sq4dx1hafdl67ra9najqcq0-source
source root is source/ui
Running phase: patchPhase
Executing npmConfigHook
Configuring npm
Validating consistency between /build/source/ui/package-lock.json and /nix/store/2mq0dhwczf70pw9iz356ffdk6wmvi3xm-waved-ui-npm-deps/package-lock.json
Installing dependencies
npm ERR! code ENOTCACHED
npm ERR! request to https://registry.npmjs.org/wrappy failed: cache mode is 'only-if-cached' but no cached response is available.

npm ERR! Log files were not written due to an error writing to the directory: /nix/store/2mq0dhwczf70pw9iz356ffdk6wmvi3xm-waved-ui-npm-deps/_logs
npm ERR! You can rerun the command with `--loglevel=verbose` to see the logs in your terminal

ERROR: npm failed to install dependencies

Here are a few things you can try, depending on the error:
1. Set `makeCacheWritable = true`
  Note that this won't help if npm is complaining about not being able to write to the logs directory -- look above that for the actual error.
2. Set `npmFlags = [ "--legacy-peer-deps" ]`

error: builder for '/nix/store/0hj7a05h8kvj8h5zhmac393242nn4pf6-waved-ui.drv' failed with exit code 1;
       last 10 log lines:
       > npm ERR! Log files were not written due to an error writing to the directory: /nix/store/2mq0dhwczf70pw9iz356ffdk6wmvi3xm-waved-ui-npm-deps/_logs
       > npm ERR! You can rerun the command with `--loglevel=verbose` to see the logs in your terminal
       >
       > ERROR: npm failed to install dependencies
       >
       > Here are a few things you can try, depending on the error:
       > 1. Set `makeCacheWritable = true`
       >   Note that this won't help if npm is complaining about not being able to write to the logs directory -- look above that for the actual error.
       > 2. Set `npmFlags = [ "--legacy-peer-deps" ]`
       >
       For full logs, run 'nix log /nix/store/0hj7a05h8kvj8h5zhmac393242nn4pf6-waved-ui.drv'.

This seems to be an outstanding issue within npm if the packages are already cached locally and npm install is used to generate the package-lock.json : https://github.com/npm/cli/issues/4460 and https://github.com/npm/cli/issues/6301

Expected behavior

In order to assure reproducible installations, every package listed in a package-lock.json should contain a resolved and integrity field so it can be fetched from a registry.

Steps To Reproduce

Unsure how to reproduce this within npm, it seems to happily proceed without caring about the missing resolved and integrity fields. Here is a simple version to verify using nix

Install nix: https://nixos.org/download/

Copy the following into default.nix:

{ pkgs ? import
(fetchTarball {
  url = "https://github.com/NixOS/nixpkgs/archive/d3f8923899a36a1985b58804de2b0cb0862015cd.tar.gz";
  sha256 = "sha256:0m2jf3chas1dls1gk6q2dg8q9ciq8mmbxbbmpfi5hjj8wd09a5kx";
})
{ }
}:
let
version = "1.1.1";
src = pkgs.fetchFromGitHub {
owner = "h2oai";
repo = "wave";
rev = "v${version}";
sha256 = "sha256-fINuoJx7dPN613wLLzcC2aar5vz6L6qzAWm/bWgj9bo=";
};
in
pkgs.buildNpmPackage {
inherit version src;
name = "waved-ui";

sourceRoot = "${src.name}/ui";

npmDepsHash = "sha256-eKBhQsqBzdC7znOiFeEAFNVKnbDXgVbBS23ZRPjpuuo=";

nodejs = pkgs.nodejs_18;

npmFlags = [ "--ignore-scripts" ];
}

nix-build from the same directory where default.nix exists
Result will be the aforementioned error

I'm happy to help troubleshoot or provide more info.

mturoci commented 3 months ago

Thanks for bringing this up @sarcasticadmin! I always assumed npm takes care of all the integrity stuff for us, but seems like it's not always the case.

Based on the npm issues you linked above, it seems like rm -rf node_modules package-lock.json should be enough to fill in the missing hashes. The unfortunate part is we will lose our lock file and dependencies will be upgraded. If you could find a way to add the hashes while keeping the existing package-lock, that would be awesome.

If not, once you confirm the issue on your side is resolved, feel free to make a PR (or just let me know here and I can make the PR myself).

sarcasticadmin commented 3 months ago

Let me give that a go and Ill raise a PR shortly

h2oai / wave