un-git-lfs benches/all-packages.nix

[flokli]

It seems there's some problems / recent breakage when trying to clone repositories with git-lfs enabled from a nix build.

In https://code.tvl.fyi/tree/tvix/eval, we're referring to rnix-parser from Cargo.lock, by git rev.

This seems to fail with more recent versions (>2.3) of Nix, due to some git-lfs issues:

 mg build //tvix/eval
[mg] building target //tvix/eval
Downloading benches/all-packages.nix (803 KB)
Error downloading object: benches/all-packages.nix (935f239): Smudge error: Error downloading benches/all-packages.nix (935f239a576caa554e546072772039f6efcbde2f2e6568733b8857fe476c5604): batch request: missing protocol: ""

Errors logged to '/home/flokli/.cache/nix/gitv3/1qzfb517dymd9fbs5nznalj8803yx2mzy4wzjxfr1wqfc3nj00q2/lfs/logs/20220916T124822.679316329.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: benches/all-packages.nix: smudge filter lfs failed
error: failed to extract archive (Source threw exception: error: program 'git' failed with exit code 128)

       … while fetching the input 'git+https://github.com/nix-community/rnix-parser.git?rev=97b438e34be5211a4b48aeed9cc3ded489b4d6da'

       … while evaluating the attribute 'checkout' of the derivation 'rnix'

       at /nix/store/0xy0cqivnzhiibhpm8481wyz0fd4xxbz-nixpkgs-src/pkgs/stdenv/generic/make-derivation.nix:270:7:

          269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
          270|       name =
             |       ^
          271|         let

       … while evaluating the attribute 'paths' of the derivation 'dependencies'

       at /nix/store/0xy0cqivnzhiibhpm8481wyz0fd4xxbz-nixpkgs-src/pkgs/stdenv/generic/make-derivation.nix:270:7:

          269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
          270|       name =
             |       ^
          271|         let

       … while evaluating 'outputVal'

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/builtins/to-toml.nix:41:15:

           40|
           41|   outputVal = v:
             |               ^
           42|     let

       … from call site

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/builtins/to-toml.nix:39:9:

           38|           "{ ${valsStr} }" else
           39|         outputVal v;
             |         ^
           40|

       … while evaluating 'outputValInner'

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/builtins/to-toml.nix:28:20:

           27|
           28|   outputValInner = v:
             |                    ^
           29|     let

       … from call site

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/builtins/to-toml.nix:69:42:

           68|           vals = mapAttrsToList
           69|             (k': v': "${quoteKey k'} = ${outputValInner v'}") v;
             |                                          ^
           70|           valsStr = concatStringsSep ", " vals;

       … while evaluating anonymous lambda

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/builtins/to-toml.nix:69:18:

           68|           vals = mapAttrsToList
           69|             (k': v': "${quoteKey k'} = ${outputValInner v'}") v;
             |                  ^
           70|           valsStr = concatStringsSep ", " vals;

       … from call site

       at /nix/store/0xy0cqivnzhiibhpm8481wyz0fd4xxbz-nixpkgs-src/lib/attrsets.nix:356:16:

          355|   mapAttrsToList = f: attrs:
          356|     map (name: f name attrs.${name}) (attrNames attrs);
             |                ^
          357|

       … while evaluating anonymous lambda

       at /nix/store/0xy0cqivnzhiibhpm8481wyz0fd4xxbz-nixpkgs-src/lib/attrsets.nix:356:10:

          355|   mapAttrsToList = f: attrs:
          356|     map (name: f name attrs.${name}) (attrNames attrs);
             |          ^
          357|

       … from call site

       … while evaluating anonymous lambda

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/builtins/to-toml.nix:133:10:

          132|       concatMapStringsSep "\n"
          133|         (kv: concatStringsSep "\n" (outputKeyVal kv.k kv.v))
             |          ^
          134|         (

       … from call site

       … while evaluating 'concatMapStringsSep'

       at /nix/store/0xy0cqivnzhiibhpm8481wyz0fd4xxbz-nixpkgs-src/lib/strings.nix:111:5:

          110|     # List of input strings
          111|     list: concatStringsSep sep (map f list);
             |     ^
          112|

       … from call site

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/builtins/to-toml.nix:132:7:

          131|     in
          132|       concatMapStringsSep "\n"
             |       ^
          133|         (kv: concatStringsSep "\n" (outputKeyVal kv.k kv.v))

       … while evaluating 'toTOML'

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/builtins/to-toml.nix:119:12:

          118|
          119|   toTOML = attrs:
             |            ^
          120|     assert (typeOf attrs == "set");

       … from call site

       at /nix/store/jw6ffjrxc08dqbnkmds0hpy0js3i8cn9-naersk-src/build.nix:84:19:

           83|     # and git dependencies.
           84|     cargoconfig = builtinz.toTOML {
             |                   ^
           85|       source = {

       … while evaluating the attribute 'cargoconfig' of the derivation 'tvix-eval-deps-0.1.0'

       at /nix/store/0xy0cqivnzhiibhpm8481wyz0fd4xxbz-nixpkgs-src/pkgs/stdenv/generic/make-derivation.nix:270:7:

          269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
          270|       name =
             |       ^
          271|         let

       … while evaluating the attribute 'builtDependencies' of the derivation 'tvix-eval-0.1.0'

       at /nix/store/0xy0cqivnzhiibhpm8481wyz0fd4xxbz-nixpkgs-src/pkgs/stdenv/generic/make-derivation.nix:270:7:

          269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
          270|       name =
             |       ^
          271|         let

It seems this is caused by some behaviourial change of builtins.fetch, causing the URL used for git-lfs to fetch artifacts lost in translation.

This repo has a single file using git-lfs, all-packages.nix, which is 784K. Adding it to the repo shouldn't be too big of an issue to workaround this.

Another possibility would be a recent release to crates.io, so we can fetch from there.

Summary & Motivation

Backwards-incompatible changes

Further context

[oberblastmeister]

I wonder if we should just download the file when running the benchmark.

[flokli]

That would mean the benchmark can't be run inside a sandbox without internet access. Is adding these 700K into the repo such a big deal?

[darichey]

I agree adding it to the repo would be fine. Also,

1. Is there an upstream Nix issue about this?
2. We should get a crates.io release out regardless! :)

[flokli]

I agree adding it to the repo would be fine. Also,

1. Is there an upstream Nix issue about this?

I'm unsure if it's an upstream Nix issue or not - i opened https://github.com/nix-community/naersk/issues/265 first to investigate there.

2. We should get a crates.io release out regardless! :)

Yes! 😆

[tazjin]

I'm unsure if it's an upstream Nix issue or not - i opened https://github.com/nix-community/naersk/issues/265 first to investigate there.

@flokli, you should be able to try this by just using builtins.fetchGit in nix-repl on a commit that still has the git-lfs on a machine with a newer Nix version