NixOS support (and other FHS-challenged distros)

[shazow]

Problem

Right now the shipped binaries make assumptions about where dynamically-linked libraries live. This is a problem for NixOS and other Linux distros that use less usual FHS layouts (there are dozens of us!).

1. The binaries that ship in the release are not statically linked, so there's several shared libraries that aren't loaded successfully (I suspect this can be patchelf'd in a pinch)

$ ldd forge
    linux-vdso.so.1 (0x00007ffe0b58f000)
    libssl.so.1.1 => not found
    libcrypto.so.1.1 => not found
[...]

2. forge build downloads a solc binary which suffers from the same problem.

$ forge build
compiling...
Error:
   0: "/home/shazow/.svm/0.8.2/solc-0.8.2": No such file or directory (os error 2)
[...]
$ ldd /home/shazow/.svm/0.8.2/solc-0.8.2
    linux-vdso.so.1 (0x00007ffec07b9000)
    libdl.so.2 => /nix/store/s9qbqh7gzacs7h68b2jfmn9l6q4jwfjz-glibc-2.33-59/lib/libdl.so.2 (0x00007fe833da6000)
    libm.so.6 => /nix/store/s9qbqh7gzacs7h68b2jfmn9l6q4jwfjz-glibc-2.33-59/lib/libm.so.6 (0x00007fe833c65000)
    libc.so.6 => /nix/store/s9qbqh7gzacs7h68b2jfmn9l6q4jwfjz-glibc-2.33-59/lib/libc.so.6 (0x00007fe833aa0000)
    /lib64/ld-linux-x86-64.so.2 => /nix/store/s9qbqh7gzacs7h68b2jfmn9l6q4jwfjz-glibc-2.33-59/lib64/ld-linux-x86-64.so.2 (0x00007fe834965000)

I tried using forge build --no-auto-detect but looks like it's refusing to use my local version.

$ which solc
/nix/store/5q9xn2s9pjav5fsh7qg8nj94gnj3jqdg-solc-0.8.2/bin/solc
$ forge build --no-auto-detect
compiling...
Error:
   0: "/home/shazow/.svm/0.8.2/solc-0.8.2": No such file or directory (os error 2)
[...]

Solutions

1. The low hanging fruit would be to build the shipped binaries as more statically linked than they are now. I do this for some other projects I maintain (mainly Go), it helps reduce all kinds of problems (can make it runnable on Alpine and other minimalist distros!) in exchange for larger binaries.

2. The more sustainable solution is to rely less on out-of-band binary fetching, and more on distro-friendly pre-packaging.

I understand that the current state of the project is in flux and perhaps the main demographic is people who copypasta curl-bash into their terminals (per the README), but realistically it does not need to be a big change to make this more distro packaging friendly. Even if we're not ready to start tagging releases, small things like changing the default Cargo dependencies to load specific versions rather than git main would go a long way (e.g. ethers-rs). Happy to send some PRs to this effect if it sounds desirable. :)

[onbjerg]

Unfortunately we can't statically link because of libusb. We're going to add packages on different package managers (apt, pacman, yum, brew are on my list), are there others you think we should have on our list?

[shazow]

I'd love nix support. :)

If you could include a flake.nix derivation, so that any distro that supports the nix package manager (basically everything) will be able to automagically build foundry from source in one command. This would also make it trivial to add to the nixpkgs package set as a full derivation (which would also give binary caches for free).

I started some work on this here, but still needs more work: https://github.com/shazow/foundry/pull/1

Somewhat related to https://github.com/gakonst/foundry/issues/315 but I wanted to clarify that you can have this declaration in a generic way that does not require any further maintenance (unless your build process changes substantially), and you don't need to use it to manage your releases or anything like that.

[Edit] The other option is to make a derivation that just downloads the latest binary releases, has all the correct inputs, and tries to patchelf the broken ldd's. It's not preferred but there are lots of binary-only packages that do this.

[Edit2] Worth noting that the solc problem will still be a thing, though. (And any other on-demand fetched binaries.)

[onbjerg]

I'll note this but do note that it is not top of my priority list currently. Feel free to open up a PR if you feel this is pressing, keeping in mind that we want to support releasechannels where possible (stable, nightly) and fixed versions for multiple package repositories, so maintainability is key here :smile:

[shazow]

I'll try to make it work out of band somehow, but right now none of this works on any of my devices unfortunately. So it is equally as pressing as me (or any other NixOS/etc user) wanting to use foundry. 😅

[shazow]

One more note: I compiled forge etc from source, and I'm able to reliably patch the svm-downloaded solc binaries with patchelf --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" ~/.svm/0.8.2/solc-0.8.2 for example, but unfortunately that changes the hash of the file which prompts svm to overwrite it before building again and putting me back in square one.

I'd love for there to be a place to disable that svm behaviour, or an alternative approach.

[gakonst]

@shazow all good points. You can force local solc version usage (under PATH or SOLC_PATH) with --no-autodetect when building or testing or disable them via the foundry toml.

Am supportive of making svm-rs more configurable, so that the paths issue does not manifest. cc @roynalnaruto

Further removing dynamic libraries seems hard, I've given it a lot of effort so far, but if you have any ideas for it I'd also be supportive, if we can let the user choose from static/dynamically linked to not inflate binary sizes too much or get us in violated license land.

[roynalnaruto]

@shazow Hi, if I'm not wrong, you are probably getting this error? (You could be sure by enabling tracing with RUST_LOG=ethers=trace and check if the messages are the same as here) https://github.com/gakonst/ethers-rs/blob/e0ee03328332776f5f18a52b968ea3f62df982cb/ethers-solc/src/resolver.rs#L408-L413

Which would re-fetch the solc binary and your changed dynamic loader is reset?

Could you try the SOLC_PATH with --no-auto-detect flag?

I am not sure how to configure svm-rs in order to support NixOS, but would be happy to take any suggestions from you or if you could PR to the repo?

[gakonst]

(We just fixed a bug with --no-auto-detect btw https://github.com/gakonst/foundry/pull/560)

[shazow]

Thanks, I'll give it a try soon!

[shazow]

Confirmed that the --no-auto-detect flag works as expected now.

I'm not sure if something like svm-rs is fixable for NixOS and similar distros. By nature, non-statically-linked binaries don't work. We have decent tooling for patching the dynamically linked paths, and one idea would be to allow for a hook to run on install but that modifies the hash so hash-based version detection goes out the window.

In a full Nix-ified version of this, you would specify the derivation and companion versions you want available, and nix would make sure it's available for you. If foundry has provisions for enabling this scenario (ie. allow for multiple versions of solc to be available for selection but without necessarily relying on a binary fetcher), then I think the rest can be done in the nix packaging out of band.

Would it be possible for it to be an ENV var rather than a flag with every command?

[Edit] Also would be even nicer if the solc version selection still works if present (perhaps via sub-path? or ENV var?) but it doesn't rely on hashes and fetching.

[shazow]

One more update: I made a stand-alone nix flake which uses the binary releases, it lives here for now (good chance I'll move it eventually): https://github.com/shazow/nixfiles/blob/master/flakes/foundry/flake.nix

This means that anyone who has nix installed can run nix develop git+https://github.com/shazow/nixfiles?dir=flakes/foundry to enter a shell that has forge and cast that has been patchelf'd to the current system.

A few caveats:

• As expected, it's pinned to specific release binaries/hashes (nix requires this). It would need to be updated with every release (this can be scripted).
• It doesn't work as a build input for other flakes yet, I think that's because I tossed it into my repo with my other nix configs and flakes don't like that apparently. :)

[shazow]

Ah figured out how to use my foundry flake as an input for my solidity project's devshell flake, in case this is useful for anyone:

# flake.nix
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    utils.url = "github:numtide/flake-utils";
    foundry.url = "git+https://github.com/shazow/nixfiles?dir=flakes/foundry";
  };

  outputs = { self, nixpkgs, utils, foundry }:
    utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system; };
      in {
        devShell = with pkgs; mkShell {
          buildInputs = [
            solc
            foundry.defaultPackage.${system}
          ];

          shellHook = ''
            export PS1="\e[1;33m\][dev]\e[1;34m\] \w $ \e[0m\]"
          '';
        };
      });
}

Then can be entered with nix develop

[gakonst]

Would it be possible for it to be an ENV var rather than a flag with every command?

You can do that via the foundry.toml file, by specifying a compiler version, see the config package readme!

[shazow]

Update in case anyone else is using this with nix: I refactored my foundry binaries derivation so it's much cleaner now and can be easily updated with latest nightlies using ./update.sh: https://github.com/shazow/nixfiles/tree/master/flakes/foundry

I'm considering putting this in another repo that is just a nix overlay, and could setup a Github CI to auto-update regularly, but probably won't do it unless other people are relying on this (feel free to ping if that's the case). :) For now I just run the update script when it's convenient for me.

(For non-nixers: this makes the equivalent of foundryup to be nix flake update in your respective project.)

[gakonst]

Separating it in an isolated Nix package that people can easily install would be great! Thanks for taking the time. We should then add it to the Readme

[shazow]

I think I have a working setup here: https://github.com/shazow/foundry.nix

Should auto-update daily. Let me know if anyone has problems!

I'd still like to track this issue as the solc thing is still a problem on NixOS, and also I'd like to get a working from-source build derivation once all of the dependencies have been pinned.

[sambacha]

Maybe this can be resolved upstream in the forthcoming release for solidity as they are overhauling their build pipeline/automation, https://github.com/ethereum/solidity/issues/13610

[neirenoir]

Given the Solidity pipeline cleanup does not seem to be going as fast as we would wish, would it make sense to add a compilation flag capable of configuring a hook that invoked a command or shell script after solc binary download? It's a quick and dirty fix, but it would solve this issue.

[hellwolf]

1) I use nix-ld, so this seems not a problem. And I specify solc_version in the foundry.toml so that foundry (setup using foundry.nix project by shazow) wouldn't attempt to download it all the time.

2) https://github.com/hellwolf/solc.nix this flake may help you to get different version of solc locally available. And with FOUNDRY_SOLC_VERSION environment variable you can choose solc-* binary you need; we use that in our CI pipeline.

[beeb]

I think I have a working setup here: https://github.com/shazow/foundry.nix

@shazow thank you so much for the overlay! Saved me a bunch of time and headaches trying to make one myself <3

[shazow]

@beeb Yay you're welcome!

Keep an eye on https://github.com/nix-community/ethereum.nix, we might merge into there once foundry has persistent tagged versions. Lots of other goodies there too!

[gakonst]

Is this working well enough? Should we close this? Should we highlight this better in the book @shazow ?

[shazow]

@gakonst Ideally I'd prefer to have versioned tagged releases so that we can do cached builds from source on NixOS. Right now it's fairly hacky (elf patching binaries).

I do agree it's a good idea to document this in the book. Should I add it to https://github.com/foundry-rs/book/blob/master/src/getting-started/installation.md?

[sambacha]

we will not be forgotten

[sambacha]

https://discourse.nixos.org/t/nix-ld-rs-testers-wanted/42145

"Nix-LD-RS provides a shim layer that allows users to specify the necessary libraries for each executable and improves the user experience by allowing users to easily run binaries from third-party sources and proprietary software"

Testers are wanted, @shazow , see the repo at: https://github.com/Mic92/nix-ld

[Padraic-O-Mhuiris]

I came across this thread after a lot of digging and experimenting with foundry + solc + nix. I've been having a difficult time making foundry reconcile different solc versions and have been following svm-rs path handling. I was thinking that if we emulate the svm folder structure while symlinking to the static solc.nix versions, it should be sufficient to run forge build --offline. @shazow and @hellwolf, have ye come across a solution for something like this?

{
  pkgs,
  lib,
  fetchFromGitHub,
  ...
}:

pkgs.stdenv.mkDerivation rec {
  pname = "contracts-bedrock";
  version = "1.4.0-rc.2";

  src = fetchFromGitHub {
    owner = "ethereum-optimism";
    repo = "optimism";
    rev = "op-contracts/v${version}";
    hash = "sha256-ryiXWuH3vIjOdMrKSeIeJOWd+7X+hTaDnWx/ugoUI/Q=";
    fetchSubmodules = true;
  };

  nativeBuildInputs = with pkgs; [ foundry-bin ];

  unpackPhase = ''
    cp $src/packages/contracts-bedrock/foundry.toml .
    cp -r $src/packages/contracts-bedrock/src .
    cp -r $src/packages/contracts-bedrock/test .
    cp -r $src/packages/contracts-bedrock/scripts .
    cp -r $src/packages/contracts-bedrock/lib .

    TEMP=$(mktemp -d)

    mkdir -p $TEMP/svm
    touch $TEMP/svm/.global-version

    mkdir -p $TEMP/svm/0.8.15
    ln -s ${lib.getExe pkgs.solc_0_8_15} $TEMP/svm/0.8.15/solc-0.8.15

    mkdir -p $TEMP/svm/0.8.24
    ln -s ${lib.getExe pkgs.solc_0_8_24} $TEMP/svm/0.8.24/solc-0.8.24

    mkdir -p $TEMP/svm/0.8.0
    ln -s ${lib.getExe pkgs.solc_0_8_0} $TEMP/svm/0.8.0/solc-0.8.0

    XDG_DATA_HOME=$TEMP
  '';

  buildPhase = ''
    forge build --offline
  '';

  meta = with lib; {
    description = "Optimism is Ethereum, scaled.";
    homepage = "https://optimism.io/";
    license = with licenses; [ mit ];
    platforms = [ "x86_64-linux" ];
  };
}

Unfortunately, the above is failing with errors:

Encountered invalid solc version in scripts/Artifacts.s.sol: No solc version installed that matches the version requirement: ^0.8.0
Encountered invalid solc version in scripts/ChainAssertions.sol: No solc version installed that matches the version requirement: ^0.8.0
Encountered invalid solc version in scripts/Chains.sol: No solc version installed that matches the version requirement: ^0.8.0
Encountered invalid solc version in scripts/Config.sol: No solc version installed that matches the version requirement: ^0.8.0
Encountered invalid solc version in scripts/Deploy.s.sol: No solc version installed that matches the version requirement: ^0.8.0
Encountered invalid solc version in scripts/DeployConfig.s.sol: No solc version installed that matches the version requirement: =0.8.15
Encountered invalid solc version in scripts/DeployOwnership.s.sol: No solc version installed that matches the version requirement: ^0.8

[Padraic-O-Mhuiris]

Fixed with missing export and some util functions, could probably be more polished but I think it works as a general solution

{
  pkgs,
  lib,
  fetchFromGitHub,
  ...
}:
let

  mkSolcSvmInstallation =
    solcPkg:
    let
      solcBinPath = lib.getExe solcPkg;
      version = lib.lists.last (
        lib.strings.splitString "-" (lib.lists.last (lib.splitString "/" solcBinPath))
      );
    in
    ''
      mkdir -p $XDG_DATA_HOME/svm/${version}
      ln -s ${solcBinPath} $XDG_DATA_HOME/svm/${version}/solc-${version}
    '';

  mkSolcSvmDataDir =
    solcPkgs:
    let
      solcInstalls = lib.strings.concatLines (lib.lists.forEach solcPkgs mkSolcSvmInstallation);
    in
    ''
      export XDG_DATA_HOME=$(mktemp -d)
      mkdir -p $XDG_DATA_HOME/svm
      touch $XDG_DATA_HOME/svm/.global-version
      ${solcInstalls}
    '';
in
pkgs.stdenv.mkDerivation rec {
  pname = "contracts-bedrock";
  version = "1.4.0-rc.2";

  src = fetchFromGitHub {
    owner = "ethereum-optimism";
    repo = "optimism";
    rev = "op-contracts/v${version}";
    hash = "sha256-ryiXWuH3vIjOdMrKSeIeJOWd+7X+hTaDnWx/ugoUI/Q=";
    fetchSubmodules = true;
  };

  nativeBuildInputs = with pkgs; [ foundry-bin ];

  unpackPhase = ''
    cp $src/packages/contracts-bedrock/foundry.toml .
    cp -r $src/packages/contracts-bedrock/src .
    cp -r $src/packages/contracts-bedrock/test .
    cp -r $src/packages/contracts-bedrock/scripts .
    cp -r $src/packages/contracts-bedrock/lib .

    ${mkSolcSvmDataDir (
      with pkgs;
      [
        solc_0_5_17
        solc_0_8_15
        solc_0_8_19
        solc_0_8_24
        solc_0_8_25
      ]
    )}
  '';

  buildPhase = ''
    forge build --offline
  '';

  installPhase = ''
    mkdir -p $out
    cp foundry.toml $out/
    cp -r src $out/
    cp -r test $out/
    cp -r scripts $out/
    cp -r lib $out/
    cp -r forge-artifacts $out/
    cp -r cache $out/
  '';

  meta = with lib; {
    description = "Optimism is Ethereum, scaled.";
    homepage = "https://optimism.io/";
    license = with licenses; [ mit ];
    platforms = [ "x86_64-linux" ];
  };
}

[shazow]

@Padraic-O-Mhuiris Very cool! Any interest in adding that as a helper to https://github.com/shazow/foundry.nix? Or maybe even https://github.com/nix-community/ethereum.nix

[Padraic-O-Mhuiris]

@shazow I'd love to when I have the time, maybe over the weekend, it actually could be a lot more polished where a bit more nix lib code can be used to "grep" for the pragma line in each solidity contract and use that to build the solc package list.

Also to make it even more generalised is utilise the foundry.toml to comprehend what relevant solidity files should be tangled out. There would be complications as some people may use npm/yarn/pnpm to pull in 3rd party contracts.

Additionally, I never understood why smart contract packaging/distribution is not very well standardised, between submodules and npm it just seems messy and maybe something nix can improve upon.

[zerosnacks]

Tagging as effectively blocked by https://github.com/foundry-rs/foundry/issues/3895 (which will be included as part of the 1.0 milestone)

[zerosnacks]

Hi all, would be great to get your input on this PR: https://github.com/foundry-rs/foundry/pull/9260

Would this be a valuable addition unblocking you?

[shazow]

@zerosnacks Thanks for the ping! Looks like a solid improvement for adding support to build foundry from source. I'd say #3895 is still blocking for proper upstream distribution/package management support (not just nixpkgs, but any other distribution too), but this is a worthy improvement.