nixos/kubernetes: extract module from nixpkgs into out-of-tree flake

[johanot]

Issue description

Perhaps this issue overflows with ambition, but I'd like to propose that we once and for all aim to get the current fully-managed Kubernetes module to live its life separately from nixpkgs.

My motivation behind this:

• Faster iteration in development of the separated flake in a separated repository
• Iteration on the remaining in-tree package will become decoupled from the module
• Get rid of (as of writing) 149 NixOS-options in nixpkgs
• Opening the door for a much simpler freeform-module for Kubernetes inside nixpkgs
• etc.

This also serves as a personal excuse for myself a least to get fully onboard with the flake-paradigm.

I hope this proposal can gain traction with NixOS 21.11 as target, making 21.05 the last release with the in-tree Kubernetes module.

Please let me hear your thoughts and concerns. cc @srhb @saschagrunert @zowoq @blaggacao

[srhb]

I am a big fan of this model (full disclosure, @johanot and I have discussed this out of band as we're coworkers)

I think the more ambitious options especially need to live outside of nixpkgs allowing the nixpkgs module itself to become a tightly coupled but thin wrapper to k8s itself.

I would love to see interest in this issue, so we know we're not just "going rogue" with this idea.

[saschagrunert]

We already discussed this some time ago in https://discourse.nixos.org/t/kubernetes-the-nixos-module-of-the-future/3922.

I'm still in favor of this approach, so +1 from my side. :blush:

[johanot]

Thanks @saschagrunert! I already forgot all about that discourse thread :)

[saschagrunert]

Do you think it's possible to have the repository living within the NixOS org?

cc @vdemeester (to get possible other contributors onboard)

[srhb]

I too had forgotten that... :scream:

Anyway, to get the ball rolling, I'd like to suggest a starting approach.

Let's create a repo (we can always transfer it to nixos community afterwards) containing the simplified, barebones "nixpkgs module" of the future. That way we can start work together by simply disabling the in-tree module and using the "vendored" one. Eventually we'll have that and the "high level" module or modules, and we can merge the barebones one back into nixpkgs.

[q3k]

As a user of (significantly forked and hacked) k8s nixos machinery, my (unrequested) kneejerk reaction/opinion is:

I'm not a fan of fragmenting nixpkgs, but that's a general issue I have with flakes. But this is not a discussion fit for this issue, if flakes are now mainstream then there's no sense in me pushing back.

However, if this means we finally get to implement a clean split between 'barebones, low-level, thin wrapper modules' and 'high-level nixos machinery for opinionated clusters', then let's do this. If time permits, I would love to contribute to this new barebones set of modules, as this would let me unfork some of hscloud's kubernetes code.

[blaggacao]

/cc @adrian-gierakowski for his recent involvement in kubenix. (fyi)

I'm very much un favor of this proposal! I'd also like to see room over there for a k3s based implentation (and why not k3os based on nix modules, but we can enter those discussion later on the target repo).

also /cc @colemickens (fyi)

Name: "kubernetix"? Goes well along with "kubenix"... (Well, just pick one, let's not discuss this here 😉 )

[blaggacao]

@johanot did you make progress on this? I'm still very much interested in a NixOS/k8s.nix

[zimbatm]

Is anybody using the NixOS Kubernetes module? It doesn't seem to be fit for purpose in its current state.

Regardless of the flakes argument, there is value in using a 3rd-party repository to explore different designs. Having to keep back-compat inside of nixpkgs can become too constraining to find the right design.

[bionicles]

i'd love to use nix-shell to install kubernetes dependencies, but alas, they (kubectl, kubeadm, docker, k3s) require sudo, which doesn't work in nix shell.

nix-shell -p kubernetes sudo kubeadm init could theoretically bootstrap a kubernetes cluster in 1 line.

Anybody know a workaround to run kubernetes (or any sudo stuff) in nix-shell? (legit clusters, not local practice toys)

is NixOS required to use sudo with nix?

[blaggacao]

You can do sudo $(which kubeadm). That works.

[ThinkChaos]

Commenting here to say I use kubernetes on NixOS in my homelab nd would be happy to test the flake. My setup is pretty standard, but more testing can't hurt!

[blaggacao]

I started trying to revive kubenix. It might eventually move into some more abstractable place rather than my personal github account.

Without a split-second of doubt, I'd consider kubenix to be the right place for this module carving-out endeavour.

Please @johanot and colleagues, feel wholeheartedly welcome. I need no fame, so moving into a neutral place at the earliest convenience is already an imperative I'm aware of.

---

If I might add, I'm currently stuck at debugging flaky tests due to https://github.com/blaggacao/kubenix/issues/5 (help please!) :wink:

[AndersonTorres]

I am not exactly favourable to this approach. Nixpkgs is a mono-repo; pulling a module out of it defeats the mono-repo.

On the other hand, IIANW it is possible to create flakes inside flakes, the same way NixOS lives inside Nixpkgs. There is no necessity to create a git repo for it.

[blaggacao]

Nixpkgs is a mono-repo; pulling a module out of it defeats the mono-repo.

LDTMRT: Let's defeat the mono-repo then. :wink: Jokes aside: in my opinion, the abstract benefits of a mono-repo are not compensating the concrete pains (described by various individuals above) with the in-tree kubernetes implementation.

[AndersonTorres]

Maintaining the resulting flake in the same git repo is unfeasible?

[srhb]

I am not exactly favourable to this approach. Nixpkgs is a mono-repo; pulling a module out of it defeats the mono-repo.

To be clear, I absolutely am fan of the nixpkgs monorepo and want to preserve it. What has been painful is trying to settle on the right design while remaining reasonably backwards-compatible. In my mind that would be easier done on top of a (very slim) NixOS (in nixpkgs) module that deals basically only with rendering configuration to the various components. This "outer layer" needn't really live out of tree, but it does need different names, and it would be nice to not opt-in to a proper deprecation cycle immediately. No one has had the time to do this anyway, but the various perspectives are helpful. :)

[blaggacao]

settle on the right design

Out of curiosity, are there already ideas around how such would look like?

[bryanasdev000]

I am also a big fan of monorepo and if possible I would like to continue within nixpkgs, anyway having a host based on NixOS running would be perfect, IMHO it would be the best OS within a selfhosted / onpremise environment.

I am still learning from Nix, but regardless of the outcome of this discussion I would like to participate as much as possible making the k8s infrastructure possible on the nix.

[blaggacao]

I am also a big fan of monorepo and if possible I would like to continue within nixpkgs, anyway having a host based on NixOS running would be perfect, IMHO it would be the best OS within a selfhosted / onpremise environment.

We are pursuing this with https://github.com/divnix/devos and kubenix support is in the pipeline.

I already put the kubernetes packages in-tree on kubenix, to have a better control over version bumps, but I guess I'll just copy over the kubernetes and k3s implementations, too.

So if people can agree on that they would not oppose to make the first step towards this issue on kubenix, then I'd be more than happy. I have no intent to lock in on any solution, and if the involved parties want to have a different place, I'd happily follow suit.

What do you guys think? Do we have a gentlemen agreement?

[stale[bot]]

I marked this as stale due to inactivity. → More info

[bryanasdev000]

Up!

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/kubernetes-on-nix-community-organization/20457/11