Profpatsch
commented
3 years ago Do you have a minimal reproduction of this problem?
Open sternenseemann opened 3 years ago
Profpatsch
commented
3 years ago Do you have a minimal reproduction of this problem?
sternenseemann
commented
3 years ago A bit messy, but this is an use case we want to support (running stuff at build time) which doesn't work nicely yet with buildNodePackage. I'm pretty sure such problems also occur more classical node packages:
let
pkgs = import <nixpkgs> { };
yarn2nix = import (pkgs.fetchFromGitHub {
owner = "Profpatsch";
repo = "yarn2nix";
rev = "2eb5647049bde3301a005e6bc9f9ea330ed0e13d";
sha256 = "0g875g4mhpw9vga1fq89sfwdd0gvnzfnn8y8nkbkcrw059k970rj";
}) { };
nixLib = yarn2nix.nixLib;
packageJson = pkgs.writeText "package.json" ''
{
"name": "test",
"version": "0.1.0",
"devDependencies": {
"parcel-bundler": "^1.12.4"
}
}
'';
yarnLock = pkgs.fetchurl {
url = "https://gist.githubusercontent.com/sternenseemann/a6d2b9a30dd8c4602529459134a7a9b1/raw/147a9a3a88375147fc38bbfeee3ec4206fb35bc9/yarn.lock";
sha256 = "1nsjg0rwm7b785l03wxdi7anhwwlw58l0i3whsvb015aihi35ksr";
};
pkgSrc = pkgs.runCommandLocal "frontend" {} ''
mkdir $out
ln -s ${packageJson} $out/package.json
ln -s ${yarnLock} $out/yarn.lock
cat > $out/index.html <<EOF
<!doctype html>
<html><head><meta charset="utf-8"><title>test</title></head>
<body>
nope
<script src="main.js"></script>
</body>
</html>
EOF
cat > $out/main.js <<EOF
console.log("lol")
EOF
'';
template = nixLib.callPackageJson packageJson {};
lock = nixLib.callYarnLock yarnLock {};
in {
yarn2nixBuild =
let
tpl = template (nixLib.buildNodeDeps lock);
linked = nixLib.linkNodeDeps {
name = tpl.key.name;
dependencies = tpl.nodeBuildInputs;
};
in pkgs.stdenv.mkDerivation {
inherit (tpl) version;
pname = tpl.key.name;
src = pkgSrc;
buildPhase = ''
export PATH="${linked}/.bin:$PATH"
parcel build index.html --out-dir=dist
'';
installPhase = ''
cp dist $out
'';
};
inherit pkgSrc;
}If you run nix-build -A yarn2nixBuild, you get the following error:
building
(node:17) UnhandledPromiseRejectionWarning: Error: Cannot find module 'babel-types'
Require stack:
- /nix/store/sq6sskky5rb6x2m749s10k5cqnv475bj-parcel-bundler-1.12.4/src/Asset.js
- /nix/store/sq6sskky5rb6x2m749s10k5cqnv475bj-parcel-bundler-1.12.4/src/assets/RawAsset.js
- /nix/store/sq6sskky5rb6x2m749s10k5cqnv475bj-parcel-bundler-1.12.4/src/Parser.js
- /nix/store/sq6sskky5rb6x2m749s10k5cqnv475bj-parcel-bundler-1.12.4/src/Bundler.js
- /nix/store/sq6sskky5rb6x2m749s10k5cqnv475bj-parcel-bundler-1.12.4/index.js
- /nix/store/sq6sskky5rb6x2m749s10k5cqnv475bj-parcel-bundler-1.12.4/src/cli.js
- /nix/store/sq6sskky5rb6x2m749s10k5cqnv475bj-parcel-bundler-1.12.4/bin/cli.js
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:965:15)
at Function.Module._load (internal/modules/cjs/loader.js:841:27)
at Module.require (internal/modules/cjs/loader.js:1025:19)
at require (/nix/store/02yfal8q6vx2md651ds7x99spb5rm3v3-v8-compile-cache-2.2.0/v8-compile-cache.js:159:20)
at Object.<anonymous> (/nix/store/sq6sskky5rb6x2m749s10k5cqnv475bj-parcel-bundler-1.12.4/src/Asset.js:12:11)
at Module._compile (/nix/store/02yfal8q6vx2md651ds7x99spb5rm3v3-v8-compile-cache-2.2.0/v8-compile-cache.js:192:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
at Module.load (internal/modules/cjs/loader.js:985:32)
at Function.Module._load (internal/modules/cjs/loader.js:878:14)
at Module.require (internal/modules/cjs/loader.js:1025:19)
(node:17) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:17) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
installing
cp: cannot stat 'dist': No such file or directory
builder for '/nix/store/5nlnzyi3xbfylasrai2fjkzm0sykasp2-test-0.1.0.drv' failed with exit code 1
error: build of '/nix/store/5nlnzyi3xbfylasrai2fjkzm0sykasp2-test-0.1.0.drv' failedThis is because of a weird traversal parcel does to discover other packages. If you use yarn however, it works:
fish> cp --no-preserve=ownership,mode -r (nix-build yarn2nix-transitive.nix -A pkgSrc) test
fish> cd test && nix-shell -p yarn
nix-shell $ yarn install
nix-shell $ yarn run parcel build --out-dir=dist index.html
# works without a nasty stack traceBelow are a directory listing of the node_modules generated by yarn2nix and yarn respectively.
Note that yarn doesn't even use symlinks. I was told that they used to do that, but even symlinking everything into one directory breaks some packages.
Profpatsch
commented
3 years ago Okay, hm, seems like we might want two flags, then, or just bite the bullet and always copy the complete transitive closure, which makes the whole thing slow.
On btrfs the cp is fast (that is copy-on-write) if you use reflink=always, but will fail on other file systems.
Do you know if yarn links the combined transitive closure of dev and runtime dependencies?
Hmm, okay, so I’d propose having two flags
{ linkTransitiveClosure ? true
, linkingMode ? "symlink" }since the symlinking the transitive closure at every stage shouldn’t be too much overhead, whereas actually copying it leads to considerable duplication and makes the build slower than necessary.
Which means those few completely braindead packages fail until you flip linkingMode to "copy" (which we should implement with reflink=auto), but that’s a good trade-off I would say.
sternenseemann
commented
3 years ago Do you know if yarn links the combined transitive closure of dev and runtime dependencies?
I think so:
Development dependencies are very much like regular dependencies except that they only matter for local packages. Packages fetched from remote registries such as npm will not be able to access their development dependencies, but packages installed from local sources (such as workspaces or the portal: protocol) will. — Source
Hmm, okay, so I’d propose having two flags
Yeah that was roughly my idea, too. reflink=auto seems like a good idea, too.
sternenseemann
commented
3 years ago It seems to me that this function implements the building of node_modules. I haven't yet looked into it too much, though.
It seems like we need to link transitive dependencies into the top-level
node_modulesdirectory because JavaScript's import routines are all kinds of broken.The solution is to implement a simple “link transitive dependencies” subcommand for
node-package-toolwhich optionally can copy the dependencies (in case symlinks are a problem for a particular package).What remains to be found out is the precise algorithm yarn uses for building
node_modules, mainly how we can solve different versions of packages being present at the same level of the dependency tree.