search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.84k stars 13.92k forks source link

feat(npm): buildNpmPackage to aling with NPM specification and implementations for copying files #219673

Open dzmitry-lahoda opened 1 year ago

dzmitry-lahoda commented 1 year ago

Describe the bug

I have tried to buildNpmPackage of https://github.com/paritytech/zombienet and failed.

Steps To Reproduce

Steps to reproduce the behavior:

  1. Clone https://github.com/paritytech/zombienet/pull/772
  2. Remove postBuild and postInstall from flake-module.nix
  3. nix run

Expected behavior

NPM builder does

  1. Omitting the field will make it default to ["*"], which means it will include all files. i
  2. main copied according spec
  3. Nix to issue warning in case of default files field or * (100% nix does not need whole NPM bloat)
dz@dz-pc-11:~/github.com/paritytech/zombienet$ nix run
warning: Git tree '/home/dz/github.com/paritytech/zombienet' is dirty
Usage: zombienet [options] [command]

Options:
  -c, --spawn-concurrency <concurrency>  Number of concurrent spawning process to launch, default
                                         is 1
  -p, --provider <provider>              Override provider to use (choices: "podman",
                                         "kubernetes", "native")

Screenshots

dz@dz-pc-11:~/github.com/paritytech/zombienet$ nix run
warning: Git tree '/home/dz/github.com/paritytech/zombienet' is dirty
node:internal/modules/cjs/loader:444
      throw err;
      ^

Error: Cannot find module '/nix/store/xiz0d5wmfnqyaq2321a3cgsr7p9wikq3-zombienet/lib/node_modules/zombienet/node_modules/@zombienet/orchestrator/dist/index.js'. Please verify that the package.json has a valid "main" entry
    at tryPackage (node:internal/modules/cjs/loader:436:19)
    at Module._findPath (node:internal/modules/cjs/loader:678:18)
    at Module._resolveFilename (node:internal/modules/cjs/loader:1061:27)
    at Module._load (node:internal/modules/cjs/loader:920:27)
    at Module.require (node:internal/modules/cjs/loader:1141:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at Object.<anonymous> (/nix/store/xiz0d5wmfnqyaq2321a3cgsr7p9wikq3-zombienet/lib/node_modules/zombienet/packages/cli/dist/cli.js:40:24)
    at Module._compile (node:internal/modules/cjs/loader:1254:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Module.load (node:internal/modules/cjs/loader:1117:32) {
  code: 'MODULE_NOT_FOUND',
  path: '/nix/store/xiz0d5wmfnqyaq2321a3cgsr7p9wikq3-zombienet/lib/node_modules/zombienet/node_modules/@zombienet/orchestrator/package.json',
  requestPath: '@zombienet/orchestrator'
}

Node.js v18.14.2

Additional context

This PR https://github.com/paritytech/zombienet/pull/772

Builds and copies dist files directly.

So it is workaround

Notify maintainers

@yu-re-ka @SuperSandro2000 @winterqt @happysalada @lilyinstarlight

Metadata

dz@dz-pc-11:~/github.com/paritytech/zombienet$ nix-shell -p nix-info --run "nix-info -m"
these 40 paths will be fetched (63.99 MiB download, 283.78 MiB unpacked):
  /nix/store/0q9hm42fapihzj1d64nxqmbml7fpb2d6-binutils-2.38
  /nix/store/2kz1pihzg1jfif46mdm917xmj6r9xyz6-diffutils-3.8
  /nix/store/2n3f8fryi27z893gd4g0g3bndhrkwnh8-attr-2.5.1
  /nix/store/31sbr22rbjw7k5dn2lc9bnf9ikyjlj4z-linux-headers-5.17
  /nix/store/3fw5n1g3bb925hfll7fj7x94bd0q6k0r-gnutar-1.34
  /nix/store/3kay4x5ifixc2c48qraq7zx9v34h6g2c-ed-1.18
  /nix/store/5ywhaxrasrp3pj72dnyy40mimx13fi0p-xz-5.2.5-bin
  /nix/store/62zsi1vvzhy09p32x28vya6x8hr2zg7a-readline-8.1p2
  /nix/store/65fgbv1iimi1r0bmz8dqgnhqz3mvjn8h-xz-5.2.5
  /nix/store/65v2c245h5qa9mpc7dxhqkfjinl6phx0-gcc-11.3.0
  /nix/store/8636bkd1gg5s8675ipb9wf98wrc5mdpw-patch-2.7.6
  /nix/store/8mhaj6yvvb7rq0kl5xmg6wl9myxvs804-gcc-11.3.0-lib
  /nix/store/93s58la8ws925xfv1661c5m1fnaaghjs-bash-interactive-5.1-p16-doc
  /nix/store/9zm6br2ri10a0b71dll2wrim5bnhg6b6-bash-5.1-p16
  /nix/store/agzy5cxd8ajmfbfksx2mbcr34wddzj6l-expand-response-params
  /nix/store/b9zc25gbziqc9qb0x6y4rsnjq92qqpid-acl-2.3.1
  /nix/store/bjhfs0gqi3p5zswg7r9bxjyn0iywq79g-glibc-2.34-210-dev
  /nix/store/bwjszg53cazbdqp17755kwv2gyilqx6i-stdenv-linux
  /nix/store/c5myz1zs9bsaq3y4s0rcxgkzb11irwra-gnumake-4.3
  /nix/store/fwc0m6fsccir8zznkl7m8iznv5wsjznw-bash-interactive-5.1-p16
  /nix/store/g71bpmj5ydqfc041qq9qdq5n5n8hm4zr-zlib-1.2.12
  /nix/store/j25abvpcbappy74w23l8lfcz7gkrsjhy-findutils-4.9.0
  /nix/store/jkv35nzx1nh8g0d5p94k8i4zjg8j8i37-nix-info
  /nix/store/la1kclnz5yr7yxhh73fjlpshbpfn0mkm-bzip2-1.0.6.0.2
  /nix/store/n5ypzvl7dijcg24isyngvw8fx0ri6hff-binutils-wrapper-2.38
  /nix/store/n7208v30hf3z4sz6127947vmzpfl46nb-bzip2-1.0.6.0.2-bin
  /nix/store/n845kl3915ri14zlkqq7wf3n3xabmgal-patchelf-0.14.5
  /nix/store/nrvhb0yvawiqgrwbmbfmhjrmy934hhs5-gzip-1.12
  /nix/store/p7pqs5c4zfc4y977p626zch11msmmpj8-gnugrep-3.7
  /nix/store/pm51x83kz65sx6lad29yrics9ny0lk4b-pcre-8.45
  /nix/store/pmh9q9k0g9s189v0iqrxpdp8j1g77gmd-gawk-5.1.1
  /nix/store/qarssrazji0q9xp80xg8shsm2crckfr0-coreutils-9.0
  /nix/store/rvgp96bwfmgz4b163flszfaygmhx8wl4-glibc-2.34-210-bin
  /nix/store/sb1wx6xz9yj05d13w5lpdljspllarbim-ncurses-6.3-p20220507
  /nix/store/v6szn6fczjbn54h7y40aj7qjijq7j6dc-glibc-2.34-210
  /nix/store/w3y3f195vndqvz7nx4bv6w3n0cyl6slv-bash-interactive-5.1-p16-info
  /nix/store/x0r01jbia3zjpxrh98rwkhrdk0yd6i1z-bash-interactive-5.1-p16-man
  /nix/store/x8mymrkpsmpwyvqssjbwsq851kscf1kw-bash-interactive-5.1-p16-dev
  /nix/store/ykcrnkiicqg1pwls9kgnmf0hd9qjqp4x-gcc-wrapper-11.3.0
  /nix/store/yyg26p5j2mrjwpkbk1djh4nxlsm2p4rw-gnused-4.8
copying path '/nix/store/93s58la8ws925xfv1661c5m1fnaaghjs-bash-interactive-5.1-p16-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/w3y3f195vndqvz7nx4bv6w3n0cyl6slv-bash-interactive-5.1-p16-info' from 'https://cache.nixos.org'...
copying path '/nix/store/x0r01jbia3zjpxrh98rwkhrdk0yd6i1z-bash-interactive-5.1-p16-man' from 'https://cache.nixos.org'...
copying path '/nix/store/v6szn6fczjbn54h7y40aj7qjijq7j6dc-glibc-2.34-210' from 'https://cache.nixos.org'...
copying path '/nix/store/31sbr22rbjw7k5dn2lc9bnf9ikyjlj4z-linux-headers-5.17' from 'https://cache.nixos.org'...
copying path '/nix/store/2n3f8fryi27z893gd4g0g3bndhrkwnh8-attr-2.5.1' from 'https://cache.nixos.org'...
copying path '/nix/store/9zm6br2ri10a0b71dll2wrim5bnhg6b6-bash-5.1-p16' from 'https://cache.nixos.org'...
copying path '/nix/store/b9zc25gbziqc9qb0x6y4rsnjq92qqpid-acl-2.3.1' from 'https://cache.nixos.org'...
copying path '/nix/store/la1kclnz5yr7yxhh73fjlpshbpfn0mkm-bzip2-1.0.6.0.2' from 'https://cache.nixos.org'...
copying path '/nix/store/qarssrazji0q9xp80xg8shsm2crckfr0-coreutils-9.0' from 'https://cache.nixos.org'...
copying path '/nix/store/n7208v30hf3z4sz6127947vmzpfl46nb-bzip2-1.0.6.0.2-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/2kz1pihzg1jfif46mdm917xmj6r9xyz6-diffutils-3.8' from 'https://cache.nixos.org'...
copying path '/nix/store/3kay4x5ifixc2c48qraq7zx9v34h6g2c-ed-1.18' from 'https://cache.nixos.org'...
copying path '/nix/store/agzy5cxd8ajmfbfksx2mbcr34wddzj6l-expand-response-params' from 'https://cache.nixos.org'...
copying path '/nix/store/j25abvpcbappy74w23l8lfcz7gkrsjhy-findutils-4.9.0' from 'https://cache.nixos.org'...
copying path '/nix/store/pmh9q9k0g9s189v0iqrxpdp8j1g77gmd-gawk-5.1.1' from 'https://cache.nixos.org'...
copying path '/nix/store/8mhaj6yvvb7rq0kl5xmg6wl9myxvs804-gcc-11.3.0-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/rvgp96bwfmgz4b163flszfaygmhx8wl4-glibc-2.34-210-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/c5myz1zs9bsaq3y4s0rcxgkzb11irwra-gnumake-4.3' from 'https://cache.nixos.org'...
copying path '/nix/store/bjhfs0gqi3p5zswg7r9bxjyn0iywq79g-glibc-2.34-210-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/yyg26p5j2mrjwpkbk1djh4nxlsm2p4rw-gnused-4.8' from 'https://cache.nixos.org'...
copying path '/nix/store/3fw5n1g3bb925hfll7fj7x94bd0q6k0r-gnutar-1.34' from 'https://cache.nixos.org'...
copying path '/nix/store/nrvhb0yvawiqgrwbmbfmhjrmy934hhs5-gzip-1.12' from 'https://cache.nixos.org'...
copying path '/nix/store/sb1wx6xz9yj05d13w5lpdljspllarbim-ncurses-6.3-p20220507' from 'https://cache.nixos.org'...
copying path '/nix/store/8636bkd1gg5s8675ipb9wf98wrc5mdpw-patch-2.7.6' from 'https://cache.nixos.org'...
copying path '/nix/store/n845kl3915ri14zlkqq7wf3n3xabmgal-patchelf-0.14.5' from 'https://cache.nixos.org'...
copying path '/nix/store/pm51x83kz65sx6lad29yrics9ny0lk4b-pcre-8.45' from 'https://cache.nixos.org'...
copying path '/nix/store/62zsi1vvzhy09p32x28vya6x8hr2zg7a-readline-8.1p2' from 'https://cache.nixos.org'...
copying path '/nix/store/p7pqs5c4zfc4y977p626zch11msmmpj8-gnugrep-3.7' from 'https://cache.nixos.org'...
copying path '/nix/store/fwc0m6fsccir8zznkl7m8iznv5wsjznw-bash-interactive-5.1-p16' from 'https://cache.nixos.org'...
copying path '/nix/store/jkv35nzx1nh8g0d5p94k8i4zjg8j8i37-nix-info' from 'https://cache.nixos.org'...
copying path '/nix/store/x8mymrkpsmpwyvqssjbwsq851kscf1kw-bash-interactive-5.1-p16-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/65fgbv1iimi1r0bmz8dqgnhqz3mvjn8h-xz-5.2.5' from 'https://cache.nixos.org'...
copying path '/nix/store/g71bpmj5ydqfc041qq9qdq5n5n8hm4zr-zlib-1.2.12' from 'https://cache.nixos.org'...
copying path '/nix/store/5ywhaxrasrp3pj72dnyy40mimx13fi0p-xz-5.2.5-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/0q9hm42fapihzj1d64nxqmbml7fpb2d6-binutils-2.38' from 'https://cache.nixos.org'...
copying path '/nix/store/65v2c245h5qa9mpc7dxhqkfjinl6phx0-gcc-11.3.0' from 'https://cache.nixos.org'...
copying path '/nix/store/n5ypzvl7dijcg24isyngvw8fx0ri6hff-binutils-wrapper-2.38' from 'https://cache.nixos.org'...
copying path '/nix/store/ykcrnkiicqg1pwls9kgnmf0hd9qjqp4x-gcc-wrapper-11.3.0' from 'https://cache.nixos.org'...
copying path '/nix/store/bwjszg53cazbdqp17755kwv2gyilqx6i-stdenv-linux' from 'https://cache.nixos.org'...
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.90.1-microsoft-standard-WSL2, Ubuntu, 20.04.5 LTS (Focal Fossa), nobuild`
 - multi-user?: `no`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.10.3`
 - channels(dz): `"nixpkgs-22.05-darwin, rust-overlay"`
 - nixpkgs: `/home/dz/.nix-defexpr/channels/nixpkgs`
dzmitry-lahoda commented 1 year ago

Will donate equivalent of 123 USD in crypto for this issue solved.

I tried several NPM solutions hard during several weeks, read all what happened during years.

We do not really have at all a reasonable way to build NPM from real world now.

Is it possible to replace homegrown npm vendor fetch to some third party maintained by npm community?

lilyinstarlight commented 1 year ago

So I just tested this and the only reason it does not work, is because the dist files are not specified as files to install in package.json: https://docs.npmjs.com/cli/v9/configuring-npm/package-json?v=true#files

However, we do need to be installing the file specified by the main field accoridng to that doc, so I'll make a PR that fixes that

This is not an issue that has anything to do with workspaces

lilyinstarlight commented 1 year ago

Fwiw, you could update the package.json file to fix that or you could just use a custom installPhase to work around it (just don't forget to prune)

dzmitry-lahoda commented 1 year ago

Omitting the field will make it default to ["*"], which means it will include all files. is default for files and it works when I set it.

dzmitry-lahoda commented 1 year ago

@lilyinstarlight than you for pointing to spec. updated issue to align with spec. actually after NPM will align, nix will build out of box a lot more NPM project without fuss.

dzmitry-lahoda commented 1 year ago

so real ultimate support of nix when nix will be extended with NPM fetcher which given all deps have SRI, respects that SRI so no need to do deps hash updates all the time