Open ghostbuster91 opened 10 months ago
I don't see anything obvious immediately but it's been a while since I read the paper. You might want to check https://github.com/haskell-nix/hnix/blob/master/src/Nix/Effects/Derivation.hs as well
Hi! was in a hurry, so sorry for terse response.
Have you managed to figure it out? It's pretty cool that you can use our code as a reference even as a non-Haskeller!
Btw, what's your motivation for implementing this?
Hi, no worries :)
Have you managed to figure it out?
Unfortunately, I wasn't able to make any progress. I bet that this is quite a subtle difference which makes it even harder to spot in the code.
It's pretty cool that you can use our code as a reference even as a non-Haskeller!
well, to some degree modulo my haskell skills :laughing:
I plan to ran your code against the same package and check both final and intermediate results, but not knowing haskell doesn't help my motivation these days.
Btw, what's your motivation for implementing this?
So, I want to create a build tool for scala that will piggy back on nix as much as possible.
One of my requirements is to use zinc which is an incremental compiler for scala. Then, based on the user input I will create on-the-fly a nix derivation that will feed the output of inc. compiler's previous run into the current build process. So basically I will curry the zinc state across multiple build run
invocations. To do this I need to be able to create nix-derivation with provided inputs as I don't won't to construct nix expressions pragmatically. Also the build configuration won't be written in nix language but in something else.
(compilerOutput, zincState1) = compileScala(sources, NoState)
(compilerOutput, zincState2) = compileScala(sources, zincState1)
I hope it makes sense :sweat_smile:
CC @flokli
@ghostbuster91 Two other things:
nix derivation add
where you don't need to pre-calculate the path.Re. content-addressing - yeah I heard about it and it seems that it should work. Not sure yet how much lack of hydra will be of a problem. I will need to check this.
However, since I got already quite far I wanted to finish implementing that approach. I didn't find many resources on that topic hence I figured out that I will write a blogpost documenting how this process works under the hood. So this kind of become a goal on its own :)
Re. nix derivation add
- sorry I didn't get this, could you elaborate?
@ghostbuster91 nix derivation add
is a new command that is basically nix derivation show
in reverse. It uses JSON for convenience; it should probably compute store paths too for convenience.
As I got cc'ed - In case reading another implementation might help - during the development of Tvix we reverse-engineered the output path calculation and produced some general-purpose (rust) code in nix-compat
that does the output path calculation - mostly the calculate_output_paths
and derivation_or_fod_hash
functions.
Consumers of this code are a bunch of testcases, as well as builtins.derivationStrict
.
Maybe some of that code helps you to understand where things happen differently?
@Ericson2314 thanks, I didn't know about this. I will check it out :+1:
@flokli
In case reading another implementation might help
It definitively will. I grew up on imperative code so reading rust should be easier. Thanks for the links :bowing_man:
Hi,
First, a little bit of context. I am trying to programatically generate nix derivation using scala. It turned out that I need to calculate nix-store path in order to put the drv file into the nix-store (which is a requirement for realizing it).
Because of that I started to implement minimialistic version of hnix-store in scala, so that I can calculate the nix-store output path. I tried reading The Purely Functional Software Deployment Model and the code in this repository (though I don't know much haskell), however it was enough for me to get started. Then I also found https://web.archive.org/web/20221001050043/https://comono.id/posts/2020-03-20-how-nix-instantiation-works/ which was an invaluable help.
I am at the point where I can calculate nix-store path correctly for some real derivations like
"/nix/store/dsn6vl7x1hbn1akgpxync19gpx2dzy8w-bootstrap-tools"
or more complex/nix/store/32lr8w57frc1ij5wzc3hb9ks8vzs2ms1-libffi-3.4.4.drv
.However, for some reason I cannot calculate correctly nix-store path for
"/nix/store/h8z4rypl78kwais0yim76czxjnd55dsm-python3-minimal-3.10.12"
There must be something different about this package/its inputDrvs but I fail to spot anything.
I wonder if you know any better resources about the algorithm used to calculate nix-store paths. I will list steps that I do, in the hope that maybe someone will be able to spot a mistake: (fixed hash derivation are left out for brevity)
sha256("output:out:sha256:${sha256(d)}:/nix/store:${d.env("name")}")
descriptor hashes are calculated as follows:
sha256("fixed:out:${d.hashAlgo}:${d.hash}:${d.path.get}")
sha256(derivation)
If anything I think that I might be handling multiple outputs incorrectly. If there is a derivation
A
that defines several outputs:and then we depend on such derivation in
B
:I don't change my logic for calculating
B
descriptor in terms ofA
and I do it the same way as if there was only a single outputout
both defined inB
and used inA
.Thanks in advance :bowing_man: