search

NNPDF / pineappl

PineAPPL is not an extension of APPLgrid
https://nnpdf.github.io/pineappl/
GNU General Public License v3.0
12 stars 3 forks source link

Nix shell #245

Open alecandido opened 11 months ago

alecandido commented 11 months ago

These are just a bunch of decoupled files to allow me to have a nix shell defined for PineAPPL.

My current goal with Nix is to have a fully reproducible environment for development (and it is fully reproducible because I'm committing it), and at the same time one I can tune for different needs (e.g. it is an easy way to test multiple Python versions, and maybe even Rust ones).

Eventually, I might even propose to package PineAPPL for Nix, within the same files. But this is not what I'm currently proposing, since the current files only support the dev shell.

FYI:

cschwan commented 11 months ago

I'm a bit worried about adding more files to the top directory, but nevermind that. Before merging I'd like to test Nix myself.

Are the build failures in commit 04fc920cdf7e56dfeeaba6654c1b3ef2317c38b2 unrelated? I've seen them before locally, but I don't know exactly what happens.

alecandido commented 11 months ago

I believe it should be because you updated something in master, and I'm based on an old version of master (I will rebase as soon as I have a stable connection).

The workflows should be completely unaffected by these files.

Btw, Nix is an optimal tool for workflows as well, because we can reproduce perfectly what is happening in the remote machine (but I never wrote yet my first Nix workflow, and the current ones are working, so it is not at all a priority). Up to action usage, of course, but with Nix you can limit the actions used (e.g. no need for actions to install software, since it can all be installed by Nix packages, and cached as well, since they are all marked by hashes).

alecandido commented 11 months ago

Ok, I just rebased on master (it was pretty much outdated), and I believe at this point there is no reason to fail any longer.

Nix is coming as a whole Linux distribution (NixOS, managing even your system configurations), but it is a build tool in the first place (just Nix) and a distribution of packages built with it (Nixpkgs), that guarantee reproducibility.

On top of Nix, I'm also using devenv, that is kind of a library to write Nix development shells + a CLI to manage them https://devenv.sh/ (and in particular I'm using it with flakes, https://devenv.sh/guides/using-with-flakes/, that is the new "flavor" of Nix, which adoption is debated, and surveys pointed slightly more than half the people answering using flakes). Development shells are already a Nix feature on their own. Devenv is just for extra convenience. (this is for @cschwan, since you said you want to try yourself - just trying to ease your way in)

alecandido commented 11 months ago

I'm stupid, and I forgot to switch to master when merging #247. I will fix it locally and push.

alecandido commented 11 months ago

Ok, solving the aftermath #247 has been effectively rebased and merged in master (in the sense that I did it locally), and this one is minimal again, and based on current master.

Sorry for the mess.

alecandido commented 7 months ago

@cschwan I managed to have a consistent configuration, where the Python package is working as well.

I thought I would have had to patch the LHAPDF distribution, but people contributing to Nix are making a great job at providing valuable and updated packages (they were up-to-date even with unmerged patches, so the Nix package was less broken than the official one). Interestingly enough, there is also a reproducible (hashed) distribution of the PDF sets themselves. https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/physics/lhapdf/pdf_sets.nix

I'm quite convinced this could be useful, since managing a multilanguage environment could be non-trivial. And it is also handling locally and consistently the LHAPDF dependency. Are you still worried by top-level files? Otherwise, could we merge?

Instructions to test

You don't need a whole NixOS to test, Nix is enough. It is available as a package on major distributions: https://repology.org/project/nix/versions

Once installed, test it with nix-info. Old-style commands begin with nix-, while there is an experimental (but very much used) interface with nix <subcommand>. Flakes themselves are marked as experimental (they have widespread adoption, half the community loves and use them apparently, but the other half hates - most likely that's the reason why they are still experimental).

To test, I would use:

nix develop . --impure

Most likely[*], you should add --extra-experimental-features nix-command --extra-experimental-features nix-command flakes (the old-style equivalent would be nix-shell, but I believe it requires a shell.nix file, and I didn't bother making it compatible).

(If you use direnv, it is integrated with Nix, and the shell will be enabled by changing the current directory within the pineappl repository. Not required.)

[*]: i.e. unless you or your distribution already configured to use experimental features

alecandido commented 7 months ago

I wanted to add target/prefix/bin to the shell path. But I need to extend the one defined by the shell, to keep access to all the other binaries provided by the Nix packages.

However, it is definitely possible, since the Python environment generated is working that way. But I'm not handling it manually, so I'm trying to understand how it works in https://github.com/cachix/devenv/blob/main/src/modules/languages/python.nix

The drawback is that, for the time being, the installed pineappl CLI won't be on your PATH by default.

cschwan commented 7 months ago

Are you still worried by top-level files? Otherwise, could we merge?

No, especially now that we document what all the files do in this repositories. Can you document them in https://github.com/NNPDF/pineappl/blob/master/docs/maintainers-guide.md?

It will take some time to review this, but after Turkey I'm confident I can.

alecandido commented 7 months ago

I fixed the $PATH issue, there was a simple solution.

No, especially now that we document what all the files do in this repositories. Can you document them in master/docs/maintainers-guide.md?

I didn't notice it, it's definitely a good idea. It takes me nothing to write a few lines to document them (I will link the relevant references to avoid verbose descriptions).

It will take some time to review this, but after Turkey I'm confident I can.

That would be great!

There is no special hurry for this, I made it in the first place because it was useful for myself. But now that is there, I believe it could be useful even for other people, so it would be a (tiny) pity to forget it forever.

alecandido commented 7 months ago

For my future self:

alecandido commented 5 months ago

Any chance for this PR to be merged?

alecandido commented 5 months ago

@cschwan I wanted to rebase this branch on master, but GitHub is complaining...

❯ git push --force-with-lease
Enumerating objects: 41, done.
Counting objects: 100% (41/41), done.
Delta compression using up to 20 threads
Compressing objects: 100% (26/26), done.
Writing objects: 100% (37/37), 8.03 KiB | 2.68 MiB/s, done.
Total 37 (delta 29), reused 15 (delta 10), pack-reused 0
remote: Resolving deltas: 100% (29/29), completed with 3 local objects.
remote: error: GH013: Repository rule violations found for refs/heads/nix.
remote: Review all repository rules at http://github.com/NNPDF/pineappl/rules?ref=refs%2Fheads%2Fnix
remote:
remote: - Cannot force-push to this branch
remote:
To github.com:NNPDF/pineappl.git
 ! [remote rejected] nix -> nix (push declined due to repository rule violations)
error: failed to push some refs to 'github.com:NNPDF/pineappl.git'

It is weird, because I rebased even one week ago, and it said nothing...

alecandido commented 5 months ago

I temporarily bypassed the limitation, but it was happening because of a new ruleset

https://github.com/NNPDF/pineappl/settings/rules/660032