Learning from Rust

[colinxs]

Awesome project! I just came across this.

I was wondering if you've looked at the various Rust packaging methods that don't require generating Nix representations of Project.toml/Manifest.toml like naersk and import-cargo. It seems like something similar could be done given the similarity between Project.toml/Manifest.toml & Cargo.toml/Cargo.lock.

Also, if you're interested there's now a #nix channel in the Julia Slack if you're interested

[thomasjm]

Oh nice, didn't know about that channel -- thanks!

Please see #3 for some other discussion on this. I haven't had time to study how those libraries work yet, but it does sound promising. If anyone is excited to get this moving/already an expert in the Rust version and wanted to sketch out a design, that would be welcome.

[colinxs]

Ah thanks for pointing out #3, I should've checked first :)

I'm not an expert, but import-cargo is only 116 LOC so hopefully not hard to grok. I think the general idea of import-cargo/naersk, however, is that the Cargo.lock contains the checksums needed to create a Julia package as a fixed-output derivation (and notably without having to specify a cargoSha256 like the current nixpkgs.rustPlatform.builtRustPackage, which is something that seems to be frowned upon, is difficult to work with, and can be avoided).

We have a similar story with the Manifest.toml with the exception that we have git tree sha's instead of checksums. I'm not entirely sure what the Cargo checksum is of, however. The package source or a tarball of it? checksum of the binary?

[thomasjm]

Just looking at that code:

• For the "registry" case, it seems the lockfile contains the sha256 of the tarball
• For the "git" case, the lockfile has a revision--apparently a normal git revision, not a tree hash. Using builtins.fetchGit instead of normal fetchgit is interesting, because it seems not to require a sha256. If builtins.fetchGit could accept a tree hash that would probably solve our problems.

[colinxs]

I think builtins.fetchGit accepting a tree hash makes a lot of sense. The fact that it accepts rev instead is kind of weird since someone could rebase a branch and yield the same commit sha but a different tree hash (which is why Pkg uses the tree hash in the first place). Would you like to open an issue for that?

Until then, could we convert the tree hash to a revision? There's probably a better way that doesn't require so much parsing but:

git log --pretty=raw | grep -B 1 <tree-hash> | awk 'NR==1{print $2}'

will print out the commit sha.

[colinxs]

Although I think that assumes there's only one commit with the corresponding tree hash?

[thomasjm]

Having multiple commits that match is okay. But it doesn't really get us closer to the goal of running "as a library" if you have to clone the repo first and and query it (unless you use the results to make a new derivation and do IFD??). In any case it's hard to imagine it being performant since you'd need a deep clone of all the package repos, or to find a way to figure out this mapping using remote git commands, which would require network calls I'm not sure how to encode in Nix...

The more I think about it it seems like a builtin is required, since there's seemingly no way to do it as a FOD at present. But it should be valid to do tree hashes as a builtin since it's just as reproducible as a rev.

It might be a tough sell to add or alter an existing builtin though, I think it requires touching the C++ code. I'd like to ponder a bit more before opening any issues, but feel free if you want to.

[thomasjm]

I asked in the Nix Hackers Matrix channel about adding a git tree hash mode to the git fetchers. They didn't say no haha. They pointed me to this RFC in progress that wants to do the same (as part of a larger initiative): https://github.com/NixOS/rfcs/pull/133

I'm going to close this issue and keep pursuing it as a separate project from julia2nix. If it ever comes to fruition, we can bring it over to julia2nix to make it run faster. (By itself, it won't eliminate the need for julia2nix to clone package repos as a preprocessing step unfortunately, because of artifacts.)