Test and document how to import Ngipkgs into an existing NixOS config

cleeyv commented 1 year ago

How to import all of Ngipkgs as an input to an existing NixOS configuration, in order to deploy a service alongside other services on the same virtual or physical machine. This could be added to the how-to documentation currently located in the README.

cleeyv commented 1 year ago

In my discussions with @ngi-nix/algae about their refactor of the flake for ngipkgs, @lorenzleutgeb shared the four ways that their new version of the flake allowed for outputs of ngipkgs to be consumed by third parties:

a. Consume all the packages in ngipkgs as an overlay to nixpkgs via outputs.overlays.default.

b. Consume individual packages via packages.${system}.${pname}.

c. Enable ngipkgs as an overlay to nixpkgs in a system configuration via a module, i.e. import nixosModules.default.

d. Consume specific modules defined in ngipkgs, e.g. nixosModules.pretalx or nixosModules.liberaforms.

What remains in this issue is to provide accessible documentation in the README with instructions on how these different methods can be used.

lorenzleutgeb commented 1 year ago

Since there was some uncertainty regarding the use of an extra argument ngipkgs (before #38) https://github.com/ngi-nix/ngipkgs/blob/148590dcbab6a09e2249a81de18bd496731b211e/flake.nix#L35-L39 vs. https://github.com/ngi-nix/ngipkgs/blob/08934529b4f1ba40613e939182532594237d07ec/flake.nix#L109-L118 using an overlay (after #38), I did an ad hoc test in one of my personal configurations: See https://github.com/lorenzleutgeb/nur/commit/150c39c1afe5e9dfd974c6a698201e4152281c4b (or the same thing in a Gist, just in case I delete the branch). This did work as expected.

lorenzleutgeb commented 5 months ago

Just missing documentation. Probably deserves to go in README.md.

cleeyv commented 4 months ago

I've done testing of how to import packages and module from the ngipkgs as it is currently structured.

Based on the tests, I've written up a PR of a new how-to section in the readme: https://github.com/ngi-nix/ngipkgs/pull/290

There are also a number of questions/comments in the PR. It may make sense for some of those discussions to be continued in this issue but I put them all in the PR for now.

cleeyv commented 3 months ago

@fricklerhandwerk and I had a long work session regarding this issue and my work so far in #290.

We started with the state of the README.md as a whole, leading to @fricklerhandwerk's rewrite of the intro in PR #316 that should be merged soon. We also discussed moving the section Structure of NGIpkgs (and also possibly Continuous builds section) into the Maintainers directory, which could be renamed Contributing and have the CONTRIBUTERS.md as a readme for that section, and if it is too long then split it off to separate files. I mention all of this in order to document some of outcomes of this conversation, even though they are not the main focus of this issue.

One important outcome of our discussion, which is also included in the rewrite in #316, is to identify three primary use cases for software from NGIpkgs:

- Run **standalone programs** locally with Nix
- Use **libraries or tools** to build software with Nixpkgs
- Deploy **services** to machines running NixOS

@fricklerhandwerk pointed out that the first one can easily be achieved with a simple nix run one liner, for example:

nix run github:ngi-nix/ngipkgs#atomic-cli

The second use case for building software is a new consideration and we put aside for now and will have to return to it later.

What we focused on the most, and that I have mostly worked on before and after this conversation, was the third use case of deploying services to NixOS. In a previous conversation we had decided to use QEMU as an example deployment target because it is ubiquitously available.

As a sort of prototype of deployment to QEMU I had created a "deployment repo" which is a pattern I have used repeatedly over the years I have been involved in Summer of Nix. We discussed the disadvantages of these kind of repos from a docs and UX perspective, and potential alternatives. The overall goal is to minimize the amount of switching between different contexts and medias the user has to do in order to provide the necessary custom inputs to the software so that it can successfully continue with its automated tasks. We looked at some browser-based examples of software that attempts to provide some kind of config deployment UX between a big monorepo and a user's desired system, such as MyNixOS and Geospatial Nix.

During the discussion we started exploring technical methods to integrate the necessary contents of my deployment repo directly into NGIpkgs. Afterwards, I continued developing these ideas and have arrived at a sketch of a way that it could be implemented.

The basic idea would be to move the flake.nix and the configuration.nix from the deployment repo into the /projects repo of NGIpkgs. The import lines would be added to the configuration.nix for all of the example configs from projects, and similarly all the nixosModules for the QEMU-deployable projects would be added to the flake.nix (possibly all of this could happen in the flake.nix to make it so only one file has to be edited). These imports and nixosModules would all be commented out by default, so required user interaction would to uncomment the services they want to deploy.

Once the projects directory is modified in this way, it basically becomes a self-contained "deployment directory" rather than a separate deployment repo. The missing git/bash-fu that makes this all work, which we had started to look into yesterday, is to allow a user to download a single directory from a repo rather than the whole repo. I have since figured out this part with the following script:

# clone an empty version of ngipkgs to a dir called deploy-ngipkgs:
git clone --no-checkout --depth=1 https://github.com/ngi-nix/ngipkgs.git deploy-ngipkgs &&
cd deploy-ngipkgs && 
git checkout main -- projects && # checkout only the projects directory of ngipkgs
cp -a projects/. . && # copy contents of the projects directory into the root of deploy-ngipkgs
rm -rf .git projects default.nix && # cleanup
git init && git add . # create a new git repo for tracking config changes during deployment

This script would be the first of three steps for the user. The second step would be, as outlined above, for them to modify the flake.nix and/or configuration.nix in deploy-nixpkgs to enable the projects they wish to deploy, and the third step would be to build and run the QEMU VM, building on what has already been tested in the flake-ngipkgs repo. This sequence for the docs still needs to be fully tested to confirm it will actually work.

This proposal raises a number of questions that I think are also relevant for evaluating the status of deployment of software from NGIpkgs:

How many of the 11 total projects have example configs? (4)
How many projects do not have example configs but do have tests and that could potentially be referred to for creation of an example config? (3) - (mCaptcha, Rosenpass, and Liberaforms)

Of the remaining project, 3 are exposed from nixpkgs (Flarum, Forgejo, mitmproxy), leaving Libre-SOC that doesn't have any modules or tests, and is just a combo of two individual packages.

Do we want to plan to add example configs for projects that are exposed from nixpkgs? This could be a way in which ngipkgs could be useful alongside nixpkgs.
Similarly would it be possible to somehow expose in their ngipkgs project dirs the nix files for tests from projects in nixpkgs (though without a need to redundantly run them in our CI)?

We also discussed yesterday how the documentation should include some basic instructions for how to go about generating a config for a project that doesn't already have one (referring to tests, for example).

ngi-nix / ngipkgs

Test and document how to import Ngipkgs into an existing NixOS config #21