Add Simple Nix Flake

[JosephGoulden]

This is new PR to add a simple flake.nix to this project. It will allow downstream projects to easily import CLN into their system. They will be able to get new releases as soon as they are available instead of having to wait for nixpkgs to be updated. And they will be able to more easily test any changes being made in CLN.

There is an existing PR #6650 but progress has stalled with getting the Python dependencies working. I think it is best to leave that for a later date and focus on a minimal working and useful flake which I have created in this PR.

The default package is called cln, it has been copied from the one currently in nixpkgs. The derivation calls the Makefile and builds the binaries (lightningd, lightning-cli, lightning-hsmtool, reckless).

There is also a package called rust which builds the Cargo workspace at the root of the project. Binaries are then copied from here into the build directory of the cln package.

The use of git submodules makes the UX a little awkward but still works as expected with the right commands. e.g;

• nix develop will create the default shell env with the build and runtime dependencies of the cln package. Then you can call make manually and the project will compile successfully.

• nix develop .#rust will create a shell env for developing rust.

• nix build .?submodules=1 will build the default package (cln).

• nix flake check .?submodules=1 will build the cln and rust packages. Rust tests are run during the build. There are also checks to run cargo audit and nixfmt.

• If you have nix installed you can run use nix run "git+https://github.com/hashrelay/lightning?ref=flake&submodules=1#lightningd" to run lightningd without having to manually clone the repo. This make use of the flake output apps.

I've added a new .version file to source control which is required in order to make a pure nix flake (fully deterministic, reproducible builds without dependencies on external environment). It will be updated during the release by the update-versions command.

I've added a new target to the CI workflow that checks the nix package builds successfully.

[chrisguida]

Hmm, for some reason it's not working for me on Ubuntu

$ nix develop
error:
       … while updating the lock file of flake 'git+file:///home/cguida/work/lightning/elementsproject?ref=refs/heads/pr7656&rev=0fc6bdf6d11a8a2a33567e480f0f7207a0172edc'

       error: unsupported tarball input attribute 'lastModified'
(cln-meta-project-py3.12) 2024-09-16 12:45:22 cguida@cg-acer:~/work/lightning/elementsproject$ nix --version
nix (Nix) 2.17.0

got any ideas?

[chrisguida]

ok i ran sudo -i nix upgrade-nix which upgraded my nix

$ sudo -i nix upgrade-nix
[sudo] password for cguida: 
replacing old 'nix-2.17.0'
installing 'nix-2.18.5'
building '/nix/store/yrgxb0lilsz84d1m8sllqq7pf6mxn5vr-user-environment.drv'...

Now it works!!

New problem though, the Rust build is failing with a giant pile of errors:

Any ideas on that?

[JosephGoulden]

All those errors stem from the "corrupt metadata" of lazy_static. I've not seen that before, can you try deleting your .cargo/registry, or at least that crate. And run cargo clean as well.

[chrisguida]

I ran rm -rf ~/.cargo/registry and now it builds!

$ lightningd/lightningd --version
v24.08-11-g0fc6bdf-modded

Seems like maybe the dev shell should handle this though, don't you think?

[chrisguida]

nix build .?submodules=1 works too!! not sure if this would have worked if I hadn't deleted the cargo registry though.

[vincenzopalazzo]

I ran rm -rf ~/.cargo/registry and now it builds!

This does not sounds right? the all point of nix is to have a sandbox, so ~/ should not be your home

[JosephGoulden]

I ran rm -rf ~/.cargo/registry and now it builds!

This does not sounds right? the all point of nix is to have a sandbox, so ~/ should not be your home

This is because I have not written derivations specifically for the cargo builds yet. I can do that later using crane if desired. At the moment the cargo builds are done by the Makefile with the cargo nativeBuildInput. It's still an improvement on the current situation as everyone using the flake will have the same version of cargo.

[JosephGoulden]

Hi @ShahanaFarooqui I'm not sure how to progress this PR. I can see you have done the past few releases. Would you be able to comment please? In particular about the .version file I proposed.

[ShahanaFarooqui]

@JosephGoulden Sorry, I am not familiar with Nix currently, so I might not be helpful here.

But as for this PR, with @chrisguida and @vincenzopalazzo reviewing, it is already in good hands.

Regarding the .version file, my suggestion might not apply in this situation, but just wanted to propose using the existing ./version_gen.h instead of adding another file for the same purpose (if that makes sense).

[chrisguida]

@JosephGoulden I think you should see if you can incorporate crane or some other framework to make sure the cargo deps are reproducible.

Lmk if you want to pair on it, I've never built anything on crane but I have some examples that other people have written that could maybe help.

Another option would be to simply disable rust since it's only needed for grpc atm, though there is also a new rust rest plugin that will hopefully be included as a built-in soon.

If you think it will take too long, we can still merge this and maybe we can get more eyes on it :)

[chrisguida]

For instance this project uses naersk and it works really well

https://github.com/chrisguida/smaug/blob/update-upstream/flake.nix

[chrisguida]

Here's another example from @RCasatta in the sling plugin https://github.com/daywalker90/sling/commit/11abe1b9a119cc606b30a8285da2c7ba99d9449d

[JosephGoulden]

@chrisguida I added a crane derivation to build the cargo workspace. The cln-grpc binary gets copied into the plugins directory by the postInstall phase of the default package. It will be easy to add the new rest plugin when that's ready.

As you know there is lots more we could do with nix but I think this is a good start.

[JosephGoulden]

@JosephGoulden Sorry, I am not familiar with Nix currently, so I might not be helpful here.

But as for this PR, with @chrisguida and @vincenzopalazzo reviewing, it is already in good hands.

Regarding the .version file, my suggestion might not apply in this situation, but just wanted to propose using the existing ./version_gen.h instead of adding another file for the same purpose (if that makes sense).

Thanks @ShahanaFarooqui. I did consider trying to reuse version_gen.h but really I think its better to have a .version file with only the version in it. Then other tools can use that to generate their required format.

[chrisguida]

Hmm build still failing. Maybe i need to update the submodules?

[chrisguida]

Ok that seems to have worked...

Now i get

It actually does work, just prints out a weird warning about clnrest.

Anyway cool, I'm good with this as long as there's documentation somewhere to tell the user to manually update the submodules by running git submodule update (maybe --init --recursive too?)

[chrisguida]

The Rust build does seem to take very long and makes my computer really hot! But at least it works :rocket:

[JosephGoulden]

The Rust build does seem to take very long and makes my computer really hot! But at least it works 🚀

It's a release build, that's normal.

[JosephGoulden]

Ok that seems to have worked...

Now i get

It actually does work, just prints out a weird warning about clnrest.

Anyway cool, I'm good with this as long as there's documentation somewhere to tell the user to manually update the submodules by running git submodule update (maybe --init --recursive too?)

Yeah git submodules are a pain. There is a reference to it in the getting-started guide. It's not specific to nix, you need to do that however you build the project.

I could add a new section to the developer guide about nix and mention it there?

Also I added a new variable to the plugins Makefile so python plugins can easily be excluded, which will fix that warning.

[chrisguida]

It's a release build, that's normal.

I've done cln-grpc build a lot, and it's never this intense. But again, I'm happy with it for now, let's ship it!

It's not specific to nix, you need to do that however you build the project.

No, the Makefile automatically updates the submodules. I've never needed to update them manually.

But again, as long as it's documented I think we're good :)

[JosephGoulden]

hi @chrisguida will you be able to approve this then?

[chrisguida]

I'll ACK if you document the git submodule thing somewhere.

But someone else will have to merge :)

[JosephGoulden]

I'll ACK if you document the git submodule thing somewhere.

But someone else will have to merge :)

I've added some Nix documentation to the developers guide. Who can we ask to merge it then?

[chrisguida]

Looks great!

Thanks so much for putting this together @JosephGoulden !

ACK https://github.com/ElementsProject/lightning/commit/93103ed5e0b292c4f96398a3706cf970f9f8dcbd

@ShahanaFarooqui @vincenzopalazzo I think this is good to go, what do you think?

[chrisguida]

Let's close #6650 when this is merged too

[ShahanaFarooqui]

@JosephGoulden Please re-base it on master to trigger the CI again. I do not have permission to push changes to your fork.

[ShahanaFarooqui]

@JosephGoulden The CI is failing because we need changelog entry in at least one of the commits.

[JosephGoulden]

Added a change log entry.

Thanks for helping me to get this in. Feel free to tag me in any related issues or pull requests.