Package request: buildBunModule

[happysalada]

Project description

https://bun.sh/ is a great project, however we still don't have a build system for it in nixpkgs. This issue is to discuss how to solve this. Discussion is continued from an initial PR https://github.com/NixOS/nixpkgs/pull/221981

Metadata

• homepage URL: https://bun.sh/

how do build a bun project today with a nix expression (this is rough, and so can't be accepted as it into nixpkgs, but should help you get something running until we figure everything out). Currently I'm using the following to build a bun project.

        node_modules = with pkgs; stdenv.mkDerivation {
          pname = "document-search-node_modules";
          version = "0.0.1";
          impureEnvVars = lib.fetchers.proxyImpureEnvVars
            ++ [ "GIT_PROXY_COMMAND" "SOCKS_SERVER" ];
          src = ./.;
          nativeBuildInputs = [ bun ];
          buildInputs = [ nodejs-slim_latest ];
          dontConfigure = true;
          buildPhase = ''
            bun install --no-progress --frozen-lockfile
          '';
          installPhase = ''
            mkdir -p $out/node_modules

            cp -R ./node_modules $out
          '';
          outputHash = if stdenv.isLinux then "sha256-vNPLtQVDLeR+TenFM0SIQ7rminT3fOUN98obaibeEW4=" else "sha256-nkPbmg/kG8Id8nGYz2x0drarDD/1qsCPL4Dgg21tmNw=";
          outputHashAlgo = "sha256";
          outputHashMode = "recursive";
        };
        document-search = with pkgs; stdenv.mkDerivation {
          pname = "document-search";
          version = "0.0.1";
          src = ./.;
          nativeBuildInputs = [ makeBinaryWrapper ];
          buildInputs = [ bun ];

          configurePhase = ''
            runHook preConfigure

            # node modules need to be copied to substitute for reference
            # substitution step cannot be done before otherwise
            # nix complains about unallowed reference in FOD
            cp -R ${node_modules}/node_modules .
            # bun installs .bin package with a usr bin env ref to node
            # replace any ref for bin that are used
            substituteInPlace node_modules/.bin/vite \
              --replace "/usr/bin/env node" "${nodejs-slim_latest}/bin/node"

            runHook postConfigure
          '';

          env.UNSTRUCTURED_API_KEY = "REPLACE_ME_UNSTRUCTURED_API_KEY";
          env.HUGGINGFACE_API_TOKEN = "REPLACE_ME_HUGGINGFACE_API_TOKEN";

          buildPhase = ''
            runHook preBuild

            bun run build

            runHook postBuild
          '';

          installPhase = ''
            runHook preInstall

            mkdir -p $out/bin
            ln -s ${node_modules}/node_modules $out
            cp -R ./build/* $out

            makeBinaryWrapper ${bun}/bin/bun $out/bin/document-search \
              # bun is referenced naked in the package.json generated script
              --prefix PATH : ${lib.makeBinPath [ bun ]} \
              --add-flags "run --prefer-offline --no-install --cwd $out start"

            runHook postInstall
          '';
        };

keep in mind this is just an example (it's a sveltekit project using vite, so your mileage may vary).

Challenges ahead.

• I'm not 100% sure the bun --install command is fully reproducible. we should potentially confirm that with upstream (even with a lockfile).
• Building the node_modules directory, should we rely on bun, or should we bun install --yarn and try to parse the yarn.lock v1 file to contruct the node_modules directory. (in the above example, I'm just using bun, but I'm not sure this can be relied on).

[happysalada]

I've been using Fixed Output Derviation for almost a month now with bun, and it looks like the output of bun install is stable. I'm planning to wait until after the 23.11 release before making a PR for buildBunModules with FOD. I think we have to test it at a bigger scale next to surface potential bugs. People on bun have been super responsive so far, so if we find something, there is a good chance there is a solution.

What I still don't know about is the stability of the sha-256 for depedencies across platforms. I think it will be different on darwin and linux, just because the output of the node_modules is probably different. This will have to be verified when there is a PR.

[cdmistman]

I think it will be different on darwin and linux, just because the output of the node_modules is probably different.

do you know if this affects any yarn2nix projects?

[cdmistman]

also, were you able to verify that bun --install is fully reproducible?

[happysalada]

I dont think yarn2nix projects are affected (since what i said is bun spevific, i have never checked the internals of yarn2nix)

I couldnt verify that its fully reproducible everytime, i just havent found a problem yet. I believe asking more people to test might surface some problems. For my package on darwin i havent had a problem yet.

[happysalada]

I'm monitoring issues in the bun repository and there are still a lof of issues open about how bun doesn't generate the correct nodes_modules. I'll keep an eye on this, and when it's more stable, I think the solution in this thread is not out of the question.

[nyabinary]

Any updates?

[happysalada]

I'm making some tests with a small package https://github.com/NixOS/nixpkgs/pull/283643 this should be abstractable into some functions later for convenience, I'm going to try to make sure that bun installation from lockfile is actually reproducible. The one thing that is going to be painful for anyone, is that every platform will have a different sha256 hash for the dependencies, that won't make any updates easy unfortunately.

[nyabinary]

I'm making some tests with a small package #283643 this should be abstractable into some functions later for convenience, I'm going to try to make sure that bun installation from lockfile is actually reproducible. The one thing that is going to be painful for anyone, is that every platform will have a different sha256 hash for the dependencies, that won't make any updates easy unfortunately.

Nice, this would make Dorion packaging much easier (#265771)

[Alexnortung]

I have been trying to bundle my own bun app.

I think it might be worth to use the --compile and --minify flag in the build step, so that you do not necessarily need to install all the source code.

For my app, I have the following build and install phas

  buildPhase = ''
    runHook preBuild

    ln -s ${node_modules}/node_modules ./node_modules
    bun build --compile --minify src/app.ts

    runHook postBuild
  '';

  dontFixup = true;

  installPhase = ''
    runHook preInstall

    mkdir -p $out/bin

    cp ./app $out/bin/${pname}

    runHook postInstall
  '';

Is this a good idea or is it better to use bun as a nativeBuildInput, such that it can be reused across other packages? If that is the case I still think --minify could be worth it.

But yeah, thanks for making this, it really helped me package my app :)

[Alexnortung]

I also want to mention that you could use the version and package name from the package.json file as default:

let
  packageJson = lib.importJSON "${src}/package.json";
  version = packageJson.version;
  pname = packageJson.name;
in

[happysalada]

Thanks for sharing this, i like both of your suggestions !

[Alexnortung]

Alright, I compared the output of using --compile and not using it. And it seems to be much better to use it without --compile. Here is the final build and install phase

  buildPhase = ''
    runHook preBuild

    ln -s ${node_modules}/node_modules ./
    bun build --minify --target bun src/app.ts > app.js

    runHook postBuild
  '';

  installPhase = ''
    runHook preInstall

    mkdir -p $out/bin

    cp app.js $out/app.js

    # bun is referenced naked in the package.json generated script
    makeBinaryWrapper ${bun}/bin/bun $out/bin/$pname \
      --add-flags "run --prefer-offline --no-install $out/app.js"

    runHook postInstall
  '';

I know that this is only for compiling a single entrypoint without a custom build script. But I think it is worth sharing anyway.

[Eveeifyeve]

I feel like a hook for this would be better #335534 has an issue that should add the hook to it.

[tsandrini]

Hi, thanks for the already done work, I wanted to share some of my experience while packaging a static Astro site built with bun via nix. Here is the snippet that I ended up with

{
  lib,
  flake-root,
  gitignoreSource,
  stdenv,
  bun,
  nodejs-slim_latest,
}:
let
  src = gitignoreSource flake-root;

  packageJson = lib.importJSON "${src}/package.json";
  version = packageJson.version;
  pname = packageJson.name;

  node_modules = stdenv.mkDerivation {
    pname = "${pname}_node-modules";
    inherit src version;

    nativeBuildInputs = [ bun ];
    buildInputs = [ nodejs-slim_latest ];

    dontConfigure = true;
    dontFixup = true; # patchShebangs produces illegal path references in FODs

    buildPhase = ''
      runHook preBuild

      export HOME=$TMPDIR

      bun install --no-progress --frozen-lockfile
      bun pm trust --all

      runHook postBuild
    '';

    installPhase = ''
      runHook preInstall

      mkdir -p $out/node_modules
      mv node_modules $out/

      runHook postInstall
    '';

    outputHash = if stdenv.isLinux then "sha256-FeMcqojV6pQiQm9ovARrNxwi3ZcLq/WwvfBzVdr8ktY=" else "";
    outputHashAlgo = "sha256";
    outputHashMode = "recursive";
  };
in
stdenv.mkDerivation {
  inherit pname version src;

  nativeBuildInputs = [
    node_modules
    nodejs-slim_latest
    bun
  ];

  configurePhase = ''
    runHook preConfigure

    cp -a ${node_modules}/node_modules ./node_modules
    chmod -R u+rw node_modules
    chmod -R u+x node_modules/.bin
    patchShebangs node_modules

    export HOME=$TMPDIR
    export PATH="$PWD/node_modules/.bin:$PATH"

    bun astro telemetry disable

    runHook postConfigure
  '';

  buildPhase = ''
    runHook preBuild

    bun build-prod

    runHook postBuild
  '';

  installPhase = ''
    runHook preInstall

    mkdir -p $out/var/www
    mv ./dist/* $out/var/www

    runHook postInstall
  '';
}

And some notes/issues that I faced

• FODs seem to be a great idea for packaging node_modules, however, running patchShebangs produced error: illegal path references in fixed-output derivation and since without the patched shebangs the build didn't work I was forced to copy (instead of symlinking it as per usual) the node_modules derivation and modify permissions + shebangs afterwards. This feels a bit suboptimal and partly defeats the purpose of FODs in the first place.
• Both astro/vite use cache directories inside node_modules (specifically node_modules/.astro and node_modules/.vite) during the build, which means that the node_modules directory needs to be writeable as well. Afaik there are no environment variables or flags to override this behaviour, the only thing you can do is to set cacheDir = 'some-other-dir'; from astro.config.mjs and vite.config.ts which won't be guaranteed for most projects (since the default is within node_modules as I stated earlier)
• I am not entirely sure why but I had a lot of issues running the astro executable. bun ./node_modules/.bin/astro ..., bunx astro and bun x astro didn't work at all. I played around with $PATH and $NODE_PATH and ended up with the combination above + the bun astro ... command format + modifying the node_modules/.bin folder permissions, however, I am not going to pretend that I really understand why it didn't work to begin with.
• bun (and iirc yarn, npm, and others) writes some cache related stuff to $HOME, so it might be a good idea to set HOME=$TMPDIR

[Eveeifyeve]

Hi, thanks for the already done work, I wanted to share some of my experience while packaging a static Astro site built with bun via nix. Here is the snippet that I ended up with

{

  lib,

  flake-root,

  gitignoreSource,

  stdenv,

  bun,

  nodejs-slim_latest,

}:

let

  src = gitignoreSource flake-root;

  packageJson = lib.importJSON "${src}/package.json";

  version = packageJson.version;

  pname = packageJson.name;

  node_modules = stdenv.mkDerivation {

    pname = "${pname}_node-modules";

    inherit src version;

    nativeBuildInputs = [ bun ];

    buildInputs = [ nodejs-slim_latest ];

    dontConfigure = true;

    dontFixup = true; # patchShebangs produces illegal path references in FODs

    buildPhase = ''

      runHook preBuild

      export HOME=$TMPDIR

      bun install --no-progress --frozen-lockfile

      bun pm trust --all

      runHook postBuild

    '';

    installPhase = ''

      runHook preInstall

      mkdir -p $out/node_modules

      mv node_modules $out/

      runHook postInstall

    '';

    outputHash = if stdenv.isLinux then "sha256-FeMcqojV6pQiQm9ovARrNxwi3ZcLq/WwvfBzVdr8ktY=" else "";

    outputHashAlgo = "sha256";

    outputHashMode = "recursive";

  };

in

stdenv.mkDerivation {

  inherit pname version src;

  nativeBuildInputs = [

    node_modules

    nodejs-slim_latest

    bun

  ];

  configurePhase = ''

    runHook preConfigure

    cp -a ${node_modules}/node_modules ./node_modules

    chmod -R u+rw node_modules

    chmod -R u+x node_modules/.bin

    patchShebangs node_modules

    export HOME=$TMPDIR

    export PATH="$PWD/node_modules/.bin:$PATH"

    bun astro telemetry disable

    runHook postConfigure

  '';

  buildPhase = ''

    runHook preBuild

    bun build-prod

    runHook postBuild

  '';

  installPhase = ''

    runHook preInstall

    mkdir -p $out/var/www

    mv ./dist/* $out/var/www

    runHook postInstall

  '';

}

And some notes/issues that I faced

• FODs seem to be a great idea for packaging node_modules, however, running patchShebangs produced error: illegal path references in fixed-output derivation and since without the patched shebangs the build didn't work I was forced to copy (instead of symlinking it as per usual) the node_modules derivation and modify permissions + shebangs afterwards. This feels a bit suboptimal and partly defeats the purpose of FODs in the first place.

• Both astro/vite use cache directories inside node_modules (specifically node_modules/.astro and node_modules/.vite) during the build, which means that the node_modules directory needs to be writeable as well. Afaik there are no environment variables or flags to override this behaviour, the only thing you can do is to set cacheDir = 'some-other-dir'; from astro.config.mjs and vite.config.ts which won't be guaranteed for most projects (since the default is within node_modules as I stated earlier)

• I am not entirely sure why but I had a lot of issues running the astro executable. bun ./node_modules/.bin/astro ..., bunx astro and bun x astro didn't work at all. I played around with $PATH and $NODE_PATH and ended up with the combination above + the bun astro ... command format + modifying the node_modules/.bin folder permissions, however, I am not going to pretend that I really understand why it didn't work to begin with.

• bun (and iirc yarn, npm, and others) writes some cache related stuff to $HOME, so it might be a good idea to set HOME=$TMPDIR

?

[honnip]

Bun creates a cache to node_modules/.cache/1234asdf.npm and it is completely non deterministic:

~> open /nix/store/s31ry31ca2f40a4jiy7qcv6k7rhq0p30-hollo-deps-0.2.0/node_modules/.cache/61531c4df79b6b8f.npm | hash sha256
a1641aee69c9085a688e954cad9126afa75f7f79be854a03724bbca3270e6f0e
~> open /nix/store/m2lqckcywc5ka9ncjc3arpysw0rra85h-hollo-deps-0.2.0/node_modules/.cache/61531c4df79b6b8f.npm | hash sha256
7c497722431de0c4e6241f9b9486a7acb33edfe9fd9e90a5dda1fee785980ae9

Installing with --no-cache flag will solve:

deps = {
  # "x86_64-linux" = "sha256-c2TKAaXAXbyI7h05paj2z0kkweA3y6P8veDWVyMHvEs=";
  # "x86_64-linux" = "sha256-mRdGKcJGFCNznLOdO8C1ncBF3eKxJnogss+zagNlVrg=";
  # "x86_64-linux" = "sha256-Cm2B3ROZHdm+tTlJv1e724JIos0woWamgo5MTBHjdxQ=";
  # "x86_64-linux" = "sha256-pqR7/pC7/kw8/PWbYyLn6/UN7wnxM54EwiUtKxDPEo4=";
  # "x86_64-linux" = "sha256-Xbq76NS2wsrcrEc9236sgDwvuHp/HslHtAB2Hp/vFBw=";
  # "x86_64-linux" = "sha256-lWurg7UOcCvLP8BG9C2JaEhXLxeB4fqX3ngJGRi2oac=";
  # "x86_64-linux" = "sha256-cSvvoOi8LSVyiJTE4Cuwth3k6oEfYP/iTSUebM7iFIE=";
  # "x86_64-linux" = "sha256-Tw+A6RTnpwQFYYmQ4PeeMe5dWJRXbPH9+VC+qzy2jKk="; # with --no-cache
  "x86_64-linux" = "sha256-Tw+A6RTnpwQFYYmQ4PeeMe5dWJRXbPH9+VC+qzy2jKk="; # yay
  "aarch64-linux" = "sha256-OeQK5RwqXzXPRBnzZfWJEieTAgtf0qF7yrEUgtMRikQ=";
};