search

rvl / bower2nix

Generate nix expressions to fetch bower dependencies.
Other
6 stars 10 forks source link

Upgrade bower2nix to work on modern Nix #22

Closed superboum closed 1 month ago

superboum commented 2 years ago

Recent node versions have dropped some features that were used by node2nix, more specifically by its bower-json dependency. It leads to the bug: Parsing /path/to/src/bower.json failed: ReferenceError: primordials is not defined.

Upgrading bower-json to its latest version 0.8.4 fixes this bug. Also now node generates a package-lock.json file that I have integrated to this PR. I also had to regenerate the node2nix files through your script regen-node2nix.sh that I had to update first to add support for the package-lock.json file. Reruning this file updated the 3 files it generated before: node-env.nix, node-packages.nix, supplement.nix and default.nix, it also now uses node 12 instead of node 4 by default.

I tested my patch on the bower.json file of the Cryptpad project. More specifically, this is how I have tested it:

nix-shell -E 'with import <nixpkgs> { }; mkShell { buildInputs = [(callPackage ./default.nix { }).package];}'
wget https://raw.githubusercontent.com/xwiki-labs/cryptpad/main/bower.json
bower2nix bower.json bower.nix

Fixes #20.

martinetd commented 2 years ago

hi @superboum I was just looking at cryptpad because node 12 is now marked insecure... So perhaps can cut my work short a bit: do you plan on updating it in nixpkgs?

I'd do it myself but this PR doesn't seem to work for me:

$ nix-shell -E 'with import ../nixpkgs { }; mkShell { buildInputs = [(callPackage ./default.nix { }).package];}'
these 2 derivations will be built:
  /nix/store/bnrkvl6rn50i5fznrs50pjs5dn6dp1dw-typescript-4.6.4.drv
  /nix/store/ga3q2r2gb2xp12z22gh15rqixyspyby4-bower2nix-3.3.0.drv
building '/nix/store/bnrkvl6rn50i5fznrs50pjs5dn6dp1dw-typescript-4.6.4.drv'...
unpacking sources
patching sources
configuring
no configure script, doing nothing
building
installing
unpacking source archive /nix/store/y2xm0xi4qjgzki4whw4pyi8wzd7ihjm9-typescript-4.6.4.tgz
pinpointing versions of dependencies...
patching script interpreter paths in .
./typescript/bin/tsc: interpreter directive changed from "#!/usr/bin/env node" to "/nix/store/i9x1ajwr756r2z9jnb2nhip50a3xms51-nodejs-16.15.0/bin/node"
./typescript/bin/tsserver: interpreter directive changed from "#!/usr/bin/env node" to "/nix/store/i9x1ajwr756r2z9jnb2nhip50a3xms51-nodejs-16.15.0/bin/node"
No package-lock.json file found, reconstructing...
rebuilt dependencies successfully
npm WARN old lockfile marborist:ctor Completed in 1ms
npm WARN old lockfile The package-lock.json file was created with an old version of npm,
npm WARN old lockfile so supplemental metadata must be fetched from the registry.
npm WARN old lockfile 
npm WARN old lockfile This is a one-time fix-up, please be patient...
npm WARN old lockfile 
npm ERR! code ENOTCACHEDmsill idealTree buildDepseps
npm ERR! request to https://registry.npmjs.org/@octokit%2frest failed: cache mode is 'only-if-cached' but no cached response is available.

npm ERR! A complete log of this run can be found in:
npm ERR!     /build/.npm/_logs/2022-05-23T11_35_14_952Z-debug-0.log

error: builder for '/nix/store/bnrkvl6rn50i5fznrs50pjs5dn6dp1dw-typescript-4.6.4.drv' failed with exit code 1;
       last 10 log lines:
       > npm WARN old lockfile so supplemental metadata must be fetched from the registry.
       > npm WARN old lockfile
       > npm WARN old lockfile This is a one-time fix-up, please be patient...
       > npm WARN old lockfile
       > npm ERR! code ENOTCACHEDmsill idealTree buildDepseps
       > npm ERR! request to https://registry.npmjs.org/@octokit%2frest failed: cache mode is 'only-if-cached' but no cached response is available.
       > 
       > npm ERR! A complete log of this run can be found in:
       > npm ERR!     /build/.npm/_logs/2022-05-23T11_35_14_952Z-debug-0.log
       > 
       For full logs, run 'nix log /nix/store/bnrkvl6rn50i5fznrs50pjs5dn6dp1dw-typescript-4.6.4.drv'.
error: 1 dependencies of derivation '/nix/store/ga3q2r2gb2xp12z22gh15rqixyspyby4-bower2nix-3.3.0.drv' failed to build

on nixos unstable; what version of nixpkgs are you using?

superboum commented 2 years ago

You must run this line of code in my branch, inside the repository folder.

It seems that you are running this code in the repository but not my branch (as I had the same bug with the missing package-lock.json). So, do you still have this bug with:

git clone git@github.com:superboum/bower2nix.git
cd bower2nix
git checkout bump-bower-json
nix-shell -E 'with import <nixpkgs> { }; mkShell { buildInputs = [(callPackage ./default.nix { }).package];}'
wget https://raw.githubusercontent.com/xwiki-labs/cryptpad/main/bower.json
bower2nix bower.json bower.nix

Also, note that we have our own packaging logic here: https://git.deuxfleurs.fr/Deuxfleurs/nixcfg/src/branch/main/app/cryptpad/build/default.nix and that we also use our own fork due to some bugs on cryptpad: https://github.com/xwiki-labs/cryptpad/pull/911 and https://github.com/xwiki-labs/cryptpad/pull/912

martinetd commented 2 years ago

You must run this line of code in my branch, inside the repository folder.

It seems that you are running this code in the repository but not my branch (as I had the same bug with the missing package-lock.json).

I fetched from this PR (git fetch origin refs/pull/22/head && git checkout FETCH_HEAD), just tried with your branch as some tools actually care about branch existing vs. detached head but there doesn't seem to be a difference. The package-lock.json is indeed present, but from the error message it probably is just that nixpkgs unstable uses a newer version of node than the node 12 you've used on nixpkgs unstable? It seems to use node 16.15.0 despite being explicitly setting nodejs ? pkgs."nodejs-12_x" in default.nix...

Ah, here it is, this works:

NIXPKGS_ALLOW_INSECURE=1 nix-shell -E 'with import <nixpkgs> { }; mkShell { buildInputs = [(callPackage ./default.nix { nodejs = pkgs.nodejs-12_x; }).package];}'

Note I need the allow insecure because node 12 has been deprecated on nixos unstable already, you need 14+ to work out of the box... (This also took me a little hour to build node 12...) But now bower2nix doesn't run

$ bower2nix bower.json bower-packages.nix
internal/modules/cjs/loader.js:818
  throw err;
  ^

Error: Cannot find module '/nix/store/9qqgmf9ipszr7vxnbq1mzb19pskrjmwd-bower2nix-3.3.0/lib/node_modules/bower2nix/bin/../dist/bower2nix'
Require stack:
- /nix/store/9qqgmf9ipszr7vxnbq1mzb19pskrjmwd-bower2nix-3.3.0/lib/node_modules/bower2nix/bin/bower2nix
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:815:15)
    at Function.Module._load (internal/modules/cjs/loader.js:667:27)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/nix/store/9qqgmf9ipszr7vxnbq1mzb19pskrjmwd-bower2nix-3.3.0/lib/node_modules/bower2nix/bin/bower2nix:3:1)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:60:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/nix/store/9qqgmf9ipszr7vxnbq1mzb19pskrjmwd-bower2nix-3.3.0/lib/node_modules/bower2nix/bin/bower2nix'
  ]
}

where /nix/store/9qqgmf9ipszr7vxnbq1mzb19pskrjmwd-bower2nix-3.3.0/lib/node_modules/bower2nix/ appears to be the bower2nix source tree; I'm clueless about node and ran out of time for today but I guess there might be another obvious problem somewhere.

To test what's next I've just tried using the bower.nix you generated in your repo, but then that fails with a similar "The package-lock.json file was created with an old version of npm" and non-cached metadata fetches that fail... Which I thought node2nix's reconstructLock was supposed to avoid... I'll try a bit harder later™

Regarding your patches -- right you probably are going to need your own package for a while. I'd also like to eventually get rid of the obsolete hardcoded jquery/bootstap that I don't understand why bower2nix doesn't find, but I did confirm they're needed as things stand :/ I wonder what requires them...

Anyway, thanks!

superboum commented 2 years ago

Thanks for your explanations, I understand better now. I will try to upgrade my PR ASAP to make it work for the latest stable node version.

I'd also like to eventually get rid of the obsolete hardcoded jquery/bootstap that I don't understand why bower2nix doesn't find, but I did confirm they're needed as things stand :/ I wonder what requires them...

I tried to run Cryptpad without adding them and once you open it in your browser, the JS loader is not happy and nothig load.

RaitoBezarius commented 2 years ago

If you're interested, @superboum — I reused your work and updated your PR to the latest stable Node version (≥ 16). Would you like I commit the delta to your branch or pick up my commits on your fork?

superboum commented 2 years ago

Sorry, I am a bit busy now, feel free to pick my commits on your fork, and open a new PR. I did not find enough time to do the work on myself, and don't plan to have more time in the near future, so I think it will be more efficient if I am out of the loop ^^

RaitoBezarius commented 2 years ago

Got it! @rvl are you still interested in a PR for this?

martinetd commented 2 years ago

@RaitoBezarius nice! I take it it's https://github.com/RaitoBezarius/bower2nix/tree/bump-bower-json ?

I'm getting the same kind of errors about old package-lock.json and non-cached dependency on today's nixpkgs-unstable:

$ nix-shell -E 'with import <nixpkgs> { }; mkShell { buildInputs = [(callPackage ./default.nix { }).package];}'
...
npm WARN old lockfile marborist:ctor Completed in 1ms
npm WARN old lockfile The package-lock.json file was created with an old version of npm,
npm WARN old lockfile so supplemental metadata must be fetched from the registry.
npm WARN old lockfile 
npm WARN old lockfile This is a one-time fix-up, please be patient...
npm WARN old lockfile 
npm ERR! code ENOTCACHEDmsill idealTree buildDepseps
npm ERR! request to https://registry.npmjs.org/@octokit%2frest failed: cache mode is 'only-if-cached' but no cached response is available.

I think I really just need to figure out how to regenerate package lock and that might be considered "common sense" ? I tried node i --package-lock-only in nix shell nixpkgs#nodejs nixpkgs#nodePackages.typescript (fun fact: I got node 16 but tsc from node 14...), but that didn't seem to refresh anything... Sorry, I wish I had time to look at how the node ecosystem works :(

RaitoBezarius commented 2 years ago

@martinetd indeed, maybe, I did not commit everything — let me get back to you in some days if it's okay with you

martinetd commented 2 years ago

Sure anytime :)

guaraqe commented 1 year ago

Hello, I found the same problem while trying to package trytond-sao, did anybody had success with the update?