Create a nix flake that supports both the package and the associated notebooks

[idontgetoutmuch]

Currently we have two nix flakes: one to build and test the library and one for running the Jupiter notebooks. Switching between these e.g. by changing branches or updating the flakes is a pain.

We'd like just one flake that provides a dev environment, allows us to to all the checks locally that get run via CI and also be able to run the notebook examples.

The alternative is stay with the manual solution.

[aleeusgr]

I will work on this too, you're welcome to guide me.

[idontgetoutmuch]

One of my colleagues advised:

Make a new flake which has your two flakes as inputs, then compose as needed

E.g. if you have a flake which exports packages.x86_64-linux.hello and another flake which exports packages.x86_64-linux.goodbye, you can do something like

{
  inputs.nixpkgs = "github:nixos/nixpkgs";
  inputs.a-flake = "github:you/a-flake";
  inputs.another-flake = "github:you/another-flake";
  outputs = { self, a-flake, another-flake }: {
    packages.x86_64-linux.a-conversation =
      nixpkgs.legacyPackages.x86_64-linux.runCommand "conversation" { }
      "${a-flake.packages.x86_64-linux.hello} > $out; ${another-flake.packages.x86_64-linux.goodbye} >> $out";
  };
}

[idontgetoutmuch]

The two flakes are

https://github.com/tweag/monad-bayes/blob/notebooks/flake.nix

and

https://github.com/tweag/monad-bayes/blob/master/flake.nix

With the former you can do

nix develop --accept-flake-config --system x86_64-darwin
jupyter-lab

and with the latter you can do

nix develop --accept-flake-config --system x86_64-darwin

and

nix --print-build-logs build .#pre-commit

[idontgetoutmuch]

@aleeusgr good luck - neither I nor @reubenharry have very high nix fu but we will help as best we can.

[aleeusgr]

@idontgetoutmuch thank you, awesome. me neither, but I'd love to improve.

You want to have flake.nix in root such that nix develop allows running jupyter-lab ?

[aleeusgr]

cd notebooks
nix develop --accept-flake-config --system x86_64-linux
jupyter-lab

jupyter-lab: command not found

[idontgetoutmuch]

Which branch are you on?

[aleeusgr]

I got both flakes running locally using git switch -c notebooks origin/notebooks

git diff origin/notebooks master -- flake.nix

hm,

{
  inputs.nixpkgs = "github:nixos/nixpkgs";
  # /home/alice/src/patchelf
  inputs.a-flake = "https://github.com/tweag/monad-bayes/blob/notebooks/flake.nix";
  inputs.another-flake = "https://github.com/tweag/monad-bayes/blob/master/flake.nix";

  --no idea how to write this, ask around
  outputs = { self, a-flake, another-flake }: {
    packages.x86_64-linux.a-conversation =
      nixpkgs.legacyPackages.x86_64-linux.runCommand "conversation" { }
      "${a-flake.packages.x86_64-linux.hello} > $out; ${another-flake.packages.x86_64-linux.goodbye} >> $out";
  };
}

also: https://nixos.org/manual/nix/unstable/command-ref/new-cli/nix3-flake.html

from discord:

You do not want a single devShell that supports both. You want to provide 2 separate devShells. I do not understand the rest of the question though. The proposed package will just fail to create a textfile…

[aleeusgr]

So from that quote, it seems to be a requirement, that both shells get combined? Perhaps start with 2 seperate and a third one that has inputsFrom = [ self.devShell.a self.devShell.b ]

[aleeusgr]

git commit -m 'new flake'
`pre-commit` not found.  Did you forget to activate your virtualenv?

[idontgetoutmuch]

Lol - yes you need to be inside the shell that does the pre-commit checks so probably something like nix develop with the flake from the main branch should work. EDIT: followed by git commit ... inside the shell. This should then run e.g. ormolu to check you are complying with the project standards.

[idontgetoutmuch]

@aleeusgr can you post a link to the discord? We do want a single shell. I'd be interested to find out why the respondent thinks we want two separate shells.

[aleeusgr]

https://discord.com/channels/568306982717751326/790886215532150815/1009699026381324329

Maybe I misrepresented something?

I don't think they think you need two shells, I think their opinion is that providing two shells and then combining them is the way to go.

TBH, for me this is way over my head, I plan to figure out how to compose flake outputs on something small, these two flakes are rather huge.

[idontgetoutmuch]

TBH, for me this is way over my head, I plan to figure out how to compose flake outputs on something small, these two flakes are rather huge.

That would be my strategy also

[aleeusgr]

Thank you for the opportunity to work on this.. https://github.com/aleeusgr/compose-flakes

[reubenharry]

Thanks so much for the help. Were you able to get this to work for our two flakes?

[aleeusgr]

Not yet, I didn't have an opportunity to dig into the latest advancements:

https://github.com/aleeusgr/compose-flakes/issues/1#issuecomment-1223905303

[aleeusgr]

I reached a milestone: a flake.nix that has two devShells in two sub-flakes and merges them into the main one.

next I will add notebooks flake to the inputs, reference it in the outputs and add the reference to inputsFrom for the devShell in flake.nix in the main

[aleeusgr]

added jf.url = "github:tweag/monad-bayes?ref=notebooks"; to inputs section result: nix develop gets error: [json.exception.parse_error.101] parse error at line 242, column 1: syntax error while parsing object key - invalid literal; last read: '"locked": {<U+000A><'; expected string literal

[aleeusgr]

the lockfile is broken, it contains merge conflict markers https://github.com/tweag/monad-bayes/blob/notebooks/flake.lock#L242

[aleeusgr]

hm I changed the reference to my fork and and in my fork's notebooks branch I have a clean flake.lock

I wonder why I get the same error..

solved: renamed the function call (I think collecting garbage and re-downloading dependencies will do the same)

[aleeusgr]

added some code, main compiles

        jf = inputs.jf1.devShell;
        monad-bayes-dev = pkgs.mkShell {
          inputsFrom = [monad-bayes.env jf];

however jupyter-lab returns command not found;

maybe I should reference https://github.com/aleeusgr/monad-bayes/blob/notebooks/flake.nix#L109 ?

[aleeusgr]

https://nixos.org/manual/nixpkgs/stable/

   outputs = { self, a-flake, another-flake }: {
     packages.x86_64-linux.a-conversation =
       **nixpkgs.legacyPackages.x86_64-linux.runCommand "conversation" { }**
       "${a-flake.packages.x86_64-linux.hello} > $out; ${another-flake.packages.x86_64-linux.goodbye} >> $out";
   };
}

ok, this works. now to clean, merge and create a pull request

[aleeusgr]

@idontgetoutmuch thank you for the idea and your guidance! https://github.com/tweag/monad-bayes/pull/187

[reubenharry]

@aleeusgr Thanks so much for working on this - it's extremely helpful

[aleeusgr]

Ur welcome. Looking forward to learning more about probabilistic programming with you!