search

xtruder / kubenix

Replaced by https://github.com/hall/kubenix
MIT License
300 stars 34 forks source link

Kubenix refactoring #9

Closed offlinehacker closed 1 year ago

offlinehacker commented 5 years ago

Description

Kubenix is in stage of refactoring due all the things we discovered at @gatehubnet while we were using kubenix for complex production deployments. We made kubenix public even before, so someone could reuse ideas, but it was still highly specialized tool for our needs. This refactoring will make kubenix generally available, but will also introduce breaking changes.

For work in progress branch please look into: https://github.com/xtruder/kubenix/tree/kubenix-2.0

New features

offlinehacker commented 5 years ago

Today i added initial istio pre-generated modules. Generator is really ugly, but gets a job done.

offlinehacker commented 5 years ago

Implemented a simple testing framework and initial tests.

offlinehacker commented 5 years ago

Initial support for e2e testing has been implemented.

offlinehacker commented 5 years ago

Helm has been integrated in https://github.com/xtruder/kubenix/commit/ba1144a8df204f2c1849593ff3256d6e1dac3c8f thanks to @matejc who implemented: https://github.com/matejc/helm2json example usage is here: https://github.com/xtruder/kubenix/commit/ba1144a8df204f2c1849593ff3256d6e1dac3c8f#diff-b025248827de015784990d5577d75c8eR32

gilligan commented 5 years ago

I'm trying to the 2.0 branch for a little test ride but i'm having some difficulties figuring out what the API has changed to now. In my code I currently have something like the following to generate different configurations:

{
  buildConfig = t: kubenix.buildResources { configuration = import ./configuration.nix { type = t; }; };
}

Looking at the code I tried something like..

  buildConfig = t: builtins.toJSON (kubenix.buildResources { 
    module = import ./configuration.nix { type = t; }; 
    modules = [ kubenix.k8s ];
  });

but that doesn't quite do the right thing and I have the feeling i am doing something very wrong here :sweat_smile:

offlinehacker commented 5 years ago

@gilligan i created a first example: https://github.com/xtruder/kubenix/tree/kubenix-2.0/examples/nginx-deployment Hope it helps :)

gilligan commented 5 years ago

@offlinehacker thank you so much :-)

offlinehacker commented 5 years ago

I did some refactoring on codebase, fixed a bug and added e2e test for nginx-deployment example (test.nix file). Will add more tests tomorrow, including for helm and kubenix submodules, stay tuned.

kalbasit commented 5 years ago
  • [ ] CI integration

@offlinehacker What's the plan with regards to CI integration? I'm very much interested in replacing Helm in a GitOps managed environment with KubeNix. Is that feasible?

cc @lzaldivarkt

offlinehacker commented 5 years ago

@kalbasit You are probably taking about e2e tests for kubenix 2.0, it's already implemented. It's spawns kubernetes cluster in qemu using nixos testing framework and nixos kubernetes module. Take a look here for example: https://github.com/xtruder/kubenix/blob/kubenix-2.0/examples/nginx-deployment/test.nix#L8 and README in same folder to see how you can run test. Also you can find bunch of unit and e2e tests for kubenix 2.0 here: https://github.com/xtruder/kubenix/tree/kubenix-2.0/tests

offlinehacker commented 5 years ago

@kalbasit also implementation of testing is defined in this kubenix module: https://github.com/xtruder/kubenix/blob/kubenix-2.0/modules/testing.nix

offlinehacker commented 5 years ago

As from implementation point of view kubenix 2.0 is getting more stable and i have implemented most if not all features we need, i will work now on documentation, more examples, and on refactoring of https://github.com/xtruder/kubenix-modules for kubenix 2.0

kalbasit commented 5 years ago

@kalbasit You are probably taking about e2e tests for kubenix 2.0, it's already implemented. It's spawns kubernetes cluster in qemu using nixos testing framework and nixos kubernetes module. Take a look here for example: /examples/nginx-deployment/test.nix@kubenix-2.0#L8 and README in same folder to see how you can run test. Also you can find bunch of unit and e2e tests for kubenix 2.0 here: /tests@kubenix-2.0

This is truly awesome!!

However, I was actually talking about JenkinsX and Prow for managing and testing pull requests against an application project on GitHub. JenkinX offers deployment of applications to multiple environments (dev, staging, prod) via GitOps, which are really just Git repositories where each repositories describe the state of the world on k8s using Helm. See https://jenkins-x.io/about/concepts/#architecture and https://www.weave.works/technologies/gitops/ for more information on how this works.

offlinehacker commented 5 years ago

This is work in progress refactoring of kubenix-submodules, i'm currently working on to make reusable set of submodules for kubenix 2.0: https://github.com/xtruder/kubenix-modules/pull/88

offlinehacker commented 5 years ago

@kalbasit We have similar setup, i will let you know how we integrate kubenix 2.0 and make an example of such setup

offlinehacker commented 5 years ago

I am running kubenix tests on hercules-ci: https://hercules-ci.com/github/xtruder/kubenix. I still did not manage to run e2e tests due some issues, but hopefully to resolve these soon :)

ghuntley commented 5 years ago

You are probably taking about e2e tests for kubenix 2.0, it's already implemented. It's spawns kubernetes cluster in qemu using nixos testing framework and nixos kubernetes module.

EPIC

zimbatm commented 5 years ago

Is the 2.0 base stable now? It seems to me like it could be merged into master. There are always more things to do.

zimbatm commented 4 years ago

ping :)

offlinehacker commented 4 years ago

KubeNix 2.0 branch was just merged into master. Documentation and tooling around still needs to be implemented and kubenix 2.0 submodule system might be refactored.

ghuntley commented 4 years ago

♥️

On Wed, 15 Jan 2020, 10:30 pm Jaka Hudoklin, notifications@github.com wrote:

KubeNix 2.0 branch was just merged into master. Documentation and tooling around still needs to be implemented and kubenix 2.0 submodule system might be refactored.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/xtruder/kubenix/issues/9?email_source=notifications&email_token=AAA7C6MN4DZJFAM2GB5D3TDQ536VVA5CNFSM4GWNANGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJAEXXI#issuecomment-574639069, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA7C6ODCXC7JK53UIYXHZ3Q536VVANCNFSM4GWNANGA .

kolloch commented 4 years ago

KubeNix 2.0 branch was just merged into master. Documentation and tooling around still needs to be implemented and kubenix 2.0 submodule system might be refactored.

If I started using kubenix now, what parts should I expect to break soon.

offlinehacker commented 4 years ago

Since I stopped working for company where we used kubenix, I also halted development of it, and am unsure if they will use it in the future.

The last thing I was working on was better e2e tests using kubetest python library, which I got it somehow finished, but I did not push it yet.

@kolloch What I was asking myself is if some parts were overengeneered, especially submodule system. What I was thinking if it is really needed or could use functional approach instead of declarative to define/import submodules.

blaggacao commented 3 years ago

@offlinehacker Would you be willing to cede ownership/maintenance so somebody else can take this over and develop it into the future. The future is bright (nickel!)

takeda commented 2 years ago

@blaggacao looking at forks seems like https://github.com/GTrunSec/kubenix is the most active currently. I wish @offlinehacker would just officially transfer ownership. When this project was being developed I didn't have a need for this project and now while I do the project seems to be dead :(

blaggacao commented 2 years ago

@takeda my conclusion is: don't use the module system as config validator. Use jsonschema validation. Much more interoperable.

Unfortunately, the upstream nix patch to add jsonschema support hasn't been accepted.

adrian-gierakowski commented 2 years ago

@takeda my conclusion is: don't use the module system as config validator. Use jsonschema validation. Much more interoperable.

Unfortunately, the upstream nix patch to add jsonschema support hasn't been accepted.

Are you saying you gave up on kubenix? Or are you suggesting it should be designed differently? Could you elaborate? Thanks!

offlinehacker commented 1 year ago

This repo has been deprecated, since I stopped maintaining it some time ago. There is a fork maintained by @hall available at https://github.com/hall/kubenix, that has better documentation and looks like a way further.