Questions

[happysalada]

I'm putting my questions here.

• I'm curious why you chose json format for communication. Would nix not work better in this case ? It might be easier to modify/override.

• My last question is around the lack of example. It would clarify things a lot to take a dummy example of a lockfile with a single package and go through what you would like each step to do.

• I mentioned it briefly in the chat, but there is naersk, that mentions no FOD and no code generation, I don't know how it works and if it could make sense to adopt a similar approach https://github.com/nix-community/naersk

[DavHau]

I'm curious why you chose json format for communication. Would nix not work better in this case ? It might be easier to modify/override.

For the generic lock file I think JSON or TOML is a good idea. The main reason is that it should be stupid and not contain any complex logic. I don't see the need of nix here. Also it is easier to generate JSON with external tools than generating nix code. Validation of the format is probably simpler as well.

My last question is around the lack of example. It would clarify things a lot to take a dummy example of a lockfile with a single package and go through what you would like each step to do.

I now added an example for the generic lock under ./specifications. More descriptions and examples will follow.

I mentioned it briefly in the chat, but there is naersk, that mentions no FOD and no code generation, I don't know how it works and if it could make sense to adopt a similar approach https://github.com/nix-community/naersk

I added ./docs/translators.md where I explain different methods of 2nix conversion and under which circumstances they can be used. Which exact method can be used in a specific situation strongly depends on the quality of data which is available. Sometimes more than one approach is possible and the goal of dream2nix is to provide all possible options to the user instead of hard coding individual approaches. The framework itself doesn't enforce IFD or code generation, but allows to use it for people who prefer it. Once the framework is specified it will definitely make sense to look at existing solutions like nearsk and take some code from there to make modules for dream2nix.

[DavHau]

I'm curious why you chose json format for communication. Would nix not work better in this case ? It might be easier to modify/override.

I forgot to talk about the overrides. So whatever format this generic lock is, it won't matter for overriding. The lock will be interpreted by a nix function (for example fetcher / builder). Those functions will provide an override interface. If they initially read data in form of json, toml or nix, shouldn't make any difference.

[happysalada]

I have some more comments

• In the translator you put the system x86_64-linux. I think we have to find a way around that, otherwise you will need to generate too much code to include darwin and aarch64 (imagine a package-lock.json for each system, people complaining about the volume of generated code in nix would go nuts).
• I know this is not the original purpose of this, but if we could find a way to remove code lock duplication accross different packages, we could have strong support from nixpkgs. Let me detail a little. At the moment, what is happening in nodePackages inside nixpkgs is essentially, grouping a bunch of packages together to try to have a smaller global lock than if each package had it's separate lock. Could we include something in the lock format that enables stuff to be fetched from nixpkgs directly ? Something like including a reference to a nixpkgs perhaps. I'm just thinking out loud here, but if we had a way to reduce code generation to a minimum, our prospects of being successful increase by a big margin. Happy to talk more about that tonight.

[DavHau]

In the translator you put the system x86_64-linux. I think we have to find a way around that, otherwise you will need to generate too much code to include darwin and aarch64 (imagine a package-lock.json for each system, people complaining about the volume of generated code in nix would go nuts).

Good point, I have removed the system for now. I think it's not the most important aspect to take care about yet. We can always fix subtle differences between platforms via overrides. We could also generate several lock files:

• one, containing system independent dependencies,

• plus one additional one for each platform which only contains the sources that need to be added only for that platform

  I know this is not the original purpose of this, but if we could find a way to remove code lock duplication accross different packages, we could have strong support from nixpkgs. Let me detail a little. At the moment, what is happening in nodePackages inside nixpkgs is essentially, grouping a bunch of packages together to try to have a smaller global lock than if each package had it's separate lock. Could we include something in the lock format that enables stuff to be fetched from nixpkgs directly ? Something like including a reference to a nixpkgs perhaps. I'm just thinking out loud here, but if we had a way to reduce code generation to a minimum, our prospects of being successful increase by a big margin. Happy to talk more about that tonight.

We could create a mechanism to dedup locked URLs and hashes. Though the mechanism has to be thought out well, as sharing stuff between packages usually increases maintenance overhead and can easily lead to frequent merge conflicts. Thanks to the abstract framework, we can deliver such optimizations at a later time if analysis come to the conclusion that this optimization is required.

Also the opt-in FOD feature can already be leveraged to reduce the size of locks, since it only needs to store one hash no matter how many packages are fetched. Compared to many nixpkgs FODs, the FOD approach used here should be much more reproducible, so we can use it quite risk free.