Idea: RFC: kube-rs based `helm` replacement

Motivation: I can't stand everything about helm, in particular working with templated yaml files stitched together like I'm working on some static blog site.

What I want: Infrastructure as Rust code.

Instead of describing my release/deployments as bunch of charts (yaml files), I'd like it to be a Rust crate, with Rust code that prepares list of k8s resources. In my CI I would just compile it, and then run it from some arguments to deploy all the resources. cargo run mycluster_release | kubectl apply -f - or just cargo run mycluster_release even.

The code of mycluster_release would be rather mundane, but at least everything would be typed, and generation of it can use any general purpose mechanism: if statments, loops, functions. It would be compiled and thus-typed-checked. No more silly mistakes caused by yaml. I could also encode any sanity-checks etc. as a code to enforce any rules I might need.

Helm has a lot of public charts, which is it's major strength. I'd like to be able to replicate it, by just - publishing a crate. So, mycluster_release can just add k8s_chart_postgresql dependency at certain version, and then let psql = k8s_chart_postgresql::Postresql::build().set_x(...).set_y(...).build(); or something like that.

Please let me know what do you think. You're feedback is greatly appreciated. Also - it seems kube-rs has a lot of things ready that I could reuse, but I'm having a hard time navigating it RN, so links to resources I should know about (relevant examples, structures in the documentation, etc.) would be very helpful.

Thanks!

While I am not particularly pleased with helm myself, this is not a simple thing to solve.

The setters you describe would basically have to re-implement yaml merges and yaml traversal. This would probably catch invalid yaml issues earlier, but it has some drawbacks. I'll ramble a little.

Pros

pretty much any actual programming language would provide a better editing interface than helms
you can provide a more targeted CI experience if you control the generation at the programming level

Cons

forces rust rust on chart maintainers (though using an actual language to wrap yaml is a popular idea: see tanka)
setters have to deal with the option heavy k8s-openapi (there's an issue Option reduction there though)
setters need to define what overriding means (shipcat did this with its own merge + derive crate, but it is not a silver bullet)

Non-Yaml Issues

invalid yaml issues can be caught pre-apply on CI running kubeval (and other yaml linters + yaml schema chechers)
invalid yaml issues on apply time can be limited with admission controllers (see gatekeeper)

These tools can pave over many of the inadequacies of the templating.

Previous Solutions

Babylon used shipcat to effectively abstract the yaml problem away from developers entirely by limiting it to one chart, and rigorously testing it. It provides a CLI to wrap helm cli for templating (with the idea that maybe we'd get away from helm entirely for templating as well, and it just being a temporary implementation detail). Feels like it has some similar goals to avoid the helm pain.

This was created before gatekeeper though. Having logic in rust code that you have to build run and test before you can deploy something related is a real hurdle when administrators can currently just get away with editing yaml live on github, wait for CI to pass, and merge. This is why shipcat is kind of a hybrid solution. People can tweak the chart, but it's a generally left to the experts. You might have to tweak the tool first to propagate things through.

There's definitely a case for structured tools that deal with one particular problem! Both shipcat and h2oai try to make things easier in one problem domain. But haven't seen anything that strikes me as "this is the solution" for generic kubernetes deployments.

Templating

There might be a lot more PROs than what I could think of. Generation can be made a lot easier to write, CI can be made a lot more dynamic, you can do fancy canary things at the programming level, but I think it needs to provide more than that to be worthy to switch to. And I think some of the CONs need to be ironed out first.

I honestly think the real problem is helm's templating, and a CLI similar to helm that was using tera as a templating engine would be immediately superior. ..but it's probably too late to switch to something like that when everyone's relying on the helm charts repos though.

Let me start by saying thank you for very thoughtful response. I really appreciate it. I agree with most of your points, with a major difference, that from my PoV the downsides don't apply to what I want to do.

forces rust rust on chart maintainers

Not to get too philosophical, but I am not a believer in business software re-usability. I don't want my team to spend their valuable time dealing with crappy DSLs and trying to mend often buggy and overly-complex general purpose software full of configs and abstractions of which 99% is of no use for our particular needs right now.

I believe that strong unification and reduction of 3rd party tools is the way to go, so if I was to build a team/dev shop I would just mandate Rust for everything. I can strongly recommend one of the best tech talks I've ever watched: https://www.youtube.com/watch?v=j6ow-UemzBc in which they did exactly that, just using Scala.

I would like to reuse core kubernetes (that's not something one wants to invest from scratch, and with clear well-defined API surface), and not all the overly-complex yaml-ridden general purpose baggage around it (like helm and alternatives). I don't necessarily want to manually do the laborious maintenance of the resource-type definitions if they are readily available. Everything else, I'm happy to have done internally.

setters have to deal with the option heavy k8s-openapi

setters need to define what overriding means

I'm not sure what it means.

The way I imagine the code would look like is:


fn get_default_pod_template() -> PodResource {
  // ...lots of common boilerplate and default settings 
}
// ...
let  api_pod = get_default_pod_template()
  .set_name("api")
  .set_somethingelse(...);

let api_backend_deployment = get_default_dp_template()
  .set_pod_template(api_pod)
  .set_autoscaling(true);

  ...

Majority of boilerplate would be taken care of by some templating/common bussiness-custom functions, and then per-case customizations would be made to them. Or in any other viable way. General purpose PLs hiding boring boilerplate is a bread and butter of software.

Does it seam feasible?

Having logic in rust code that you have to build run and test before you can deploy something related is a real hurdle when administrators can currently just get away with editing yaml live on github, wait for CI to pass, and merge.

I don't understand what's the hurdle exactly. Administrators can live-edit the Rust code, wait for the CI to pass and merge it. The only downside that I see is an additional compilation step, but it would pay for itself, just in the amount of problems caught by static typing, IMO.

But haven't seen anything that strikes me as "this is the solution" for generic kubernetes deployments.

I am not interested in generic, only building blocks to build a specific, minimal tools.

So... which libraries/API/tools are the closest to having a raw PodResource DeploymentResource and potentially all other resource-type types, without having to do it manually?

Ah. If you are in a position where you can go down the route of talk (and that talk is indeed great) and mandate rust then your case becomes a lot more attractive. Sorry, I wrongly assumed a more general intent, a lot of my response isn't actually super valid for your case. If everyone uses rust then the administrator problem isn't really a problem (the language unfamiliarity of administrators was essentially the main part of my argument there).

The only downside that I see is an additional compilation step, but it would pay for itself, just in the amount of problems caught by static typing, IMO.

Yeah, the compilation step slows down CI jobs a bit, but it's not a big deal. In a generic world with generic tools, you could have cached images with tools, rather than build the tools, then run them. But if you don't have any external config management outside rust then you might not need to bring in any external tooling anyway.

So... which libraries/API/tools are the closest to having a raw PodResource DeploymentResource and potentially all other resource-type types, without having to do it manually?

https://github.com/Arnavion/k8s-openapi has the structs for all these things. Is that what you are looking for?

Does it seam feasible?

Yeah, for sure. You probably don't even need to use kube if you are using kubectl, you can build straight on k8s-openapi.

Elaborating a bit on the two awkward points;

setters have to deal with the option heavy k8s-openapi

setters need to define what overriding means

The first refers to https://github.com/Arnavion/k8s-openapi/issues/72 . It's a small ergonomic issue with how the kubernetes types are presented from generated resources. If you're not trying to solve things in a super generic way, whatever. it probably just means the solution starts looking nicer at some point in the future if it gets resolved.

The second is more to do with how bits of yaml overrides other parts of yaml. I assumed you were thinking of the DSL route; i.e. replacing helm template --set somekey=somevalue with some kind of .set (then you get into the awkward territory of what happens when parts of the struct is missing (in case of nesting), dealing with defaults, what happens when you override an existing struct with an empty struct, what happens when you override a full vector with an empty vector).

Just override parts of structs with structs makes complete sense. You might have to think about some of those things down the line if you try to implement some kind of api patch mechanism on top of it (strategic patch behaviour might not completely align with the rust types, and it has limitations for CRDs), but if you're doing all application with kubectl then this is also a moot point.

Yeah, the compilation step slows down CI jobs a bit,

Still a bit immature, but https://github.com/LukeMathWalker/cargo-chef is so far my go-to tool, and I've been doing the same thing manually in the Dockerfiles for my Rust projects, coupled with https://github.com/dpc/docker-source-checksum with good results for monorepo-based orgs.

https://github.com/Arnavion/k8s-openapi has the structs for all these things. Is that what you are looking for?

Yes! Thank you!

The first refers to Arnavion/k8s-openapi#72 . It's a small ergonomic issue with how the kubernetes types are presented from generated resources.

Oh, I see. Yeah, it's small nit, as far as I can tell.

if you try to implement some kind of api patch mechanism on top of it

Just to double check: You're talking about kubectl patch / kubectl edit functionality, right? Is it being commonly used for anything other than manually hacks/quick fixes?

The second is more to do with how bits of yaml overrides other parts of yaml. I assumed you were thinking of the DSL route; i.e. replacing helm template --set somekey=somevalue (...)

Oh, I see. If I was to ever do that, I would expose properly via some config file and/or stuctopt command line options or something, but since the tool like this would not be general purpose, I would rather avoid it for anything else than absolute neccessities (like maybe specifing which deployment (prod vs staging vs test vs dev etc.).

Thanks again! (If I ever get something noticeable actually working, I'll make sure to return and give some feedback).

caching

Yeah, those are good ones!

You're talking about kubectl patch / kubectl edit functionality, right? Is it being commonly used for anything other than manually hacks/quick fixes?

Yeah those ones. Controllers running within-cluster to ensure resources stay within spec also tend to rely on patch a lot. Even kubectl uses patch under the hood to apply (unless you have serverside apply enabled).

Thanks again!

Looking forward to hearing how it goes! Best of luck.

kube-rs / kube