Cargo build without cargoSha256, use Cargo.lock instead

nmattia commented 5 years ago

Issue description

I’ve been experimenting with a new build platform that doesn’t require specifying a cargoSha256 but instead relies on the information contained in the Cargo.lock. There are some open issues, maybe someone here will have good solutions!

(note: happy to move this to discourse if needed)

The problems with the current rust platform (in particular with cargoSha256) is that it’s

unergonomic: every single dependency update requires the developer to update the cargoSha256. This means tweaking the existing checksum, re-building, copying the new hash, etc. This becomes a huge mess when lots of developers are working on the same cargo project.
not repeatable: if a developer forgets to update the hash, nix will use the old dependencies (nix store lookup by sha256) and in some cases the build may succeed nonetheless (think: security patch in a dependency, which doesn’t tinker with the crate’s interface). However a build on a different machine — where the vendored crates haven’t been cached yet — will result in a hash mismatch.

How

The idea is to leverage the “recently” added builtins.fromTOML to read the Cargo.lock in Nix directly and fetch all crates before the build even begins (I’ve done something similar for npm). The lockfile looks something like this:

[[package]]
name = "aho-corasick"
version = "0.6.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
 "memchr 2.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
...

[metadata]
"checksum aho-corasick 0.6.10 (registry+https://github.com/rust-lang/crates.io-index)" = "81ce3d38065e618af2d7b77e10c5ad9a069859b4be3c2250f674af3840d9c8a5"

Quoting naersk (the POC for this new platform):

    # XXX: the actual crate format is not documented but in practice is a
    # gzipped tar; we simply unpack it and introduce a ".cargo-checksum.json"
    # file that cargo itself uses to double check the sha256

The corollary is that the lockfile contains all the information Nix needs in order to download dependencies (at least for crates coming from crates.io, see below for git dependencies).

Current status

The POC works surprisingly well. Most libs build out of the box but there are some issues that worry me a bit.

Libs that build

(I basically went through the list of trending rust repos)

Missing

crate-level incrementality: currently all dependencies are built alongside the main crate. This is not too bad (rustc is pretty fast) but it could be improved.
cargo doesn’t register checksums for git dependencies: this is a pretty nasty issue, because fetchFromGitHub and friends require a sha256. There are some workarounds:
- use builtins.fetchgit ... which doesn’t require a sha256. The major drawbacks is that fetchgit will redownload the repo every hour and doesn’t support submodules. Moreover it’s somewhat ugly.
- use cargoSha256 for git deps ... which is somewhat work ergonomic than having a single cargoSha256 for all deps, but still not ideal
- track git dependencies outside of cargo ... e.g. “vendor” the dependencies through nix. The major drawback is that — while it’s fairly easy to do source replacement in a nix-build/derivation — it’s a pain to make sure that cargo (in a dev’s terminal) uses the same dependencies. There are some workarounds that can be implemented in a nix-shell (shellHook setting $CARGO_HOME with custom config, or shellHook that symlinks the store paths in $PWD) but it’s still very, very suboptimal.

CC @grahamc @basvandijk

stale[bot] commented 4 years ago

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

Search for maintainers and people that previously touched the related code and @ mention them in a comment.
Ask on the NixOS Discourse.
Ask on the #nixos channel on irc.freenode.net.

Hoverbear commented 4 years ago

Oh my gosh this would be so handy to have official!

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

Hoverbear commented 3 years ago

This is still important to me!

Licenser commented 3 years ago

I'd love to see this too :+1:

nerosnm commented 3 years ago

I would also still really love to see this!

nmattia commented 3 years ago

Note: naersk was just moved to https://github.com/nix-community/naersk, so I guess we're one step closer!

stale[bot] commented 2 years ago

I marked this as stale due to inactivity. → More info

Hoverbear commented 2 years ago

This is still important to me!

Stunkymonkey commented 1 year ago

fyi: #221716

NixOS / nixpkgs