helm render support

[jaxxstorm]

Is it possible to:

• render a helm chart locally (using `helm template
• modify/manipulate that helm chart (for example, patch it somehow?)
• then write it back out as yaml?

If it is, if I can get some help I can create an example for the examples repo. If not, could it be added as an option?

This is possible in Pulumi by doing something like this:

import * as k8s from "@pulumi/kubernetes";

const redis = new k8s.helm.v2.Chart("redis", {
    repo: "stable",
    chart: "redis",
    version: "3.10.0",
    values: {
        usePassword: true,
        rbac: { create: true },
    },
    transformations: [
        // Make every service private to the cluster, i.e., turn all services into ClusterIP instead of
        // LoadBalancer.
        (obj: any) => {
            if (obj.kind == "Service" && obj.apiVersion == "v1") {
                if (obj.spec && obj.spec.type && obj.spec.type == "LoadBalancer") {
                    obj.spec.type = "ClusterIP";
                }
            }
        }
    ]
});

So I'm hoping it's possible in jk, too.

[dlespiau]

Oh, pulumi has a nice way to express it! We have some of what is needed to do the above. The main missing thing is to be able to execute a command.

• I see helm template is directly invoked by pulumi. We have so far explicitly excluded executing commands because they can do arbitrary things, which conflicts with being hermetic.
• We would be able to load yaml files that have been rendered by a previous pass before jk though.
• I can see how nice the example you refer to is for consuming helm charts and wonder how we could do something similar. Especially because one could use jk as a glue layer, injecting the same variables in various helm charts and then doing global transforms on the resulting YAML files. That sounds like a nice thing to be able to do.
• We can read yaml files with std.read
• We can (deep) patch the objects and use transforms on them with mix and patch. Note we'd like to have some fairly fundamental object processing functions in the standard library sooner rather than later (See #117)
• I think, compared to the example above, it may be slightly nicer to keep the object transforms orthogonal to the Chart object
• Write the object back as yaml with std.write

I think this use case makes sense, which leaves me to wonder what would be needed to support it :) Will have to think about it a bit more.

[jaxxstorm]

Thanks for the thoughtful answer!

Couple of things:

I see helm template is directly invoked by pulumi. We have so far explicitly excluded executing commands because they can do arbitrary things, which conflicts with being hermetic. We would be able to load yaml files that have been rendered by a previous pass before jk though.

This is generally what we do with do with kr8 - something to bear in mind though is that generally the yaml comes as a stream and sometimes needs cleaning up.

I can see how nice the example you refer to is for consuming helm charts and wonder how we could do something similar The main thing that's nice from Pulumi is the transformations. I've lost count of the number of times a helm chart isn't working properly and have to patch it in some way. I think this is addresses with #117

I think this use case makes sense, which leaves me to wonder what would be needed to support it :)

It seems to be the only thing missing is some method of rendering the helm chart. Maybe it's better that this is done outside of JK and it just needs to be documented on how to do it?

[squaremo]

We've talked a bit about having "escapes" for the hermeticity, e.g., to run jk as a server by importing @jkcfg/http/server -- perhaps (allow-listed, sandboxed) exec could be one of those.

[squaremo]

Another way this could be accomplished is from the outside -- if you can read from stdin (a source of un-safety in its own way!) it could transform the documents in a YAML stream, without needing to exec itself.

[dlespiau]

Random thoughts:

• exposing stdin somehow sounds useful in its own right.
• with helm charts, I think there's value in having the jk script do the exec: this way we can have a top level script that generate a bunch of charts, enabling sharing parameters across charts and global operations on the rendered objects.
• I was thinking along the same lines as @squaremo: an std.exec function with an explicit command line argument to enabled it (or config option in a config file, most likely both). I'll write up those thoughts in a new issue when time allows, hopefully today.

[dlespiau]

@jaxxstorm I've perused your blog and it seems we all have very similar views on configuration :) If you want to talk about configuration in general or jk in particular, feel free to join the #jkcfg slack channel, I've just created one on our slack. Be warned, it's brand new and have only a few people on it :)

[sh0rez]

About Helm and hermeticity .. the helm and tiller are written in golang as well. And Tiller does not actually need to run in cluster-wide server mode.

• https://github.com/adamreese/helm-local
• https://rimusz.net/tillerless-helm/

So we could reuse the important parts of Helm to render the charts ourselves .. probably in a small standalone application (helm-shim?).

ksonnet did this back then and it worked quite well, although overall approach had it's pitfalls. So I propose the following:

• Some small application governed by jkcfg provides guarantees about rendering charts (binary pinning, linking against helm, the way outlined above)
• The stdlib gets a standardized way of talking to shims (I could imagine other kinds as well), maybe using gRPC like Terraform does? https://github.com/hashicorp/go-plugin. The shims receive an object from JS and return one (go map[string]interface{}). They shims must make sure they are deterministic at any point.

[sh0rez]

About Helm and hermeticity .. the helm and tiller are written in golang as well. And Tiller does not actually need to run in cluster-wide server mode. ...

This assumes we do not actually want to use helm for release management but install helm charts as flat .yaml manifests to the cluster, which is IMHO the better way, because I am not a big fan of moving parts. This is also the reason why I imagine jkcfg being superior to pulumi. State hurts but having no state = not losing any state!

[dlespiau]

@sh0rez yup all of the above makes sense :)

I didn't know about go-plugin, sounds interesting. I was thinking about a very simple "protocol" where plugins/subprocesses would output their results on stdout, mainly because I was only thinking about a generic std.exec that would be able to execute helm directly to do the rendering. Having a bit more to it with a plugin system would allow cleaner interaction such as better error handling and leave room for further extensions (we have some vague ideas about run-time things where the jk process would stay around and react to events).

Being able to download a specific plugin version that in turn is pinning the version of helm instead of relying on whatever version of helm is running on the machine is certainly a nice property.

We could also think about downloading the plugins automatically, jk could do so when encountering special std directives. go-getter seems nice for the implementation side of this.

food for thoughts :)

[sh0rez]

I totally agree!

IMHO we could take Terraform as a reference, as the people over there already built a production-tested, bullet proof expansion system: While Terraform provides control structures, the Providers do the heavy lifting of creating actually creating resources. But a provider is a standalone executable that is loosely coupled with Terraform and thus allows independent updates. They communicate using gRPC, implemented using above mentioned go-plugin. This is nice, because gRPC specifies a typed API using protobuf and is not locked to only GoLang but any other supported language. The providers are not bundled with Terraform application code but held ready for dynamic loading in a artifact-signing registry. As go-getter is used, this registry might actually be anything from s3 over git repo to artifactory or whatnot. This could be specified by the user or even inferred from JavaScript by creating versioned npm modules for our shims.

If desired, I can contribute the plugin system but I would need guidance for the interaction with JavaScript, as my knowledge in this field is limited to React and frontend right now.

[dlespiau]

I really don't want to be too much "stop energy" but there is one downside to the go-plugin approach: the complexity it brings.

• It's a lot of code
• We'd need to write one provider for each thing we want to support
• We'd need to maintain those providers and update them at the same rhythm as the upstream software
• What happens if the user wants to pin the helm version to a version we don't support
• We'd need to host binaries for the plugins
• We'd need jk to download and cache those binaries
• At the moment, I can't really think of a second user of this plugin interface, but I'm sure that would come in time.

We'd also want to flesh out how it would look like from a js point of view.

The alternative std.exec is a lot more straightforward. I've sketched something in issue #147. Note that those 2 ideas (std.exec and plugins) could be both implemented if needed. I'm struggling a bit imagining what else a plugin system would bring to the table, so far I have (these may well be enough to justify plugins):

• Version pinning for reproducible builds: we don't depend on the helm version being installed on the machine but download a specific version
• Automatic install our the run-time dependencies, we don't need something else to install helm for us

[squaremo]

With #149 it is possible to pipe the output of helm template through a jk script to transform it. A script that does the same transformation as in the OP looks like this:

// transform-chart.js
import std from '@jkcfg/std';

async function readAndTransform() {
  const resources = await(std.read('', { format: std.Format.YAMLStream }));
  resources.forEach((obj) => {
    if (obj === null) return;
    if (obj.apiVersion == 'v1' && obj.kind == 'Service') {
      if (obj.spec && obj.spec.type && obj.spec.type == 'LoadBalancer') {
        obj.spec.type = 'ClusterIP';
      }
    }
  });
  return resources;
}

readAndTransform().then(v => std.log(v, { format: std.Format.YAMLStream }));

helm fetch stable/redis --untar --untardir=charts/
helm template charts/redis | jk run ./transform-chart.js

Now, I wouldn't claim this is a substitute for the Pulumi code -- jk is doing just one part -- but it does demonstrate that it's fairly easy to get the same effect.

[sh0rez]

I really don't want to be too much "stop energy" but there is one downside to the go-plugin approach: the complexity it brings.

• It's a lot of code

Not sure about this.

• We'd need to write one provider for each thing we want to support
• We'd need to maintain those providers and update them at the same rhythm as the upstream software

This is true, but I believe it is far more simple to maintain upstream changes directly in-tree. And for example helm did not change it's chart format since v2 has been released. This means helm template is quite stable. The chart format is going to change in v3, but since then we probably do not need to adapt.

• What happens if the user wants to pin the helm version to a version we don't support

I think the chart rendering process of helm does not change that frequently. So it might be fine to extract helm template (which has been a standalone application before it was merged into helm) and provide it as helm-static config provider for jk. Probably just start simple and adapt if needed.

• We'd need to host binaries for the plugins

We may use GitHub Releases for this – infrastructure for free.

• We'd need jk to download and cache those binaries

go-getter does a great job at this and is already used in production at massive scale. If it is only helm(-static) for the beginning, we could ship it in the same binary to reduce complexity and load it dynamically if needed later along the way.

• At the moment, I can't really think of a second user of this plugin interface, but I'm sure that would come in time.

Something I can imagine of right know:

• Vault Provider (get secrets from vault). Vault provides far more mature secret engine that Kubernetes does. Dynamically getting secrets might not be reproducible but is still required for multi-environment, etc. Doing this on the fly using approle simplfies secures the whole CD process
• Kubernetes Provider (read any objects). This might be nice if one would like to create objects based on other objects. This could reduce the need of actual operators as a cron-job ob jk and kubectl apply would suffice to add sidecars for example.
• Legacy configuration formats that are already deployed: For example, I have a lot of jsonnet configs lying around – I don't really like these untested things but I am actively using them in my HomeLab due to a lack of alternatives. I like the idea of jk, because especially with TypeScript it allows to detect errors before deploying and building tested libraries of config code. It would be very nice to incorporate the old solutions into the new one to reduce overall complexity.

Using plugins, we could provide some guarantees of stability to the user. And jk can stay self-contained, which improves UX a lot.

We'd also want to flesh out how it would look like from a js point of view.

The alternative std.exec is a lot more straightforward. I've sketched something in issue #147. Note that those 2 ideas (std.exec and plugins) could be both implemented if needed. I'm struggling a bit imagining what else a plugin system would bring to the table, so far I have (these may well be enough to justify plugins):

• Version pinning for reproducible builds: we don't depend on the helm version being installed on the machine but download a specific version
• Automatic install our the run-time dependencies, we don't need something else to install helm for us

While the std.exec is a great first step, it provides poor integration with the target system. Think of error handling, version pinning.

Furthermore, I don't believe integration of third-party features software should happen at JavaScript level. I mainly like the idea of using JavaScript for writing the configuration, because it is a very flexible, functional language that is already bullet-proof, testable and has a great ecosystem (which jsonnet for example totally lacks). But think it should be limited by design to defining the configuration. While DRY is very nice, we still should focus on some amount of KISS. If the configuration frontend allows to much, it might quickly get overwhelming, especially to new users. So jk should provide some boundaries to maintain a low learning curve. I really do not want to start debugging my configuration generation code to find race-conditions or nasty bugs.

Idea: If the providers are not part of jk, community members or companies with internal tools could write them as well and publish them to jkcfg once stable? And there are a lot of internal, hacked solutions. If they could be incorporated into one single, flexible engine – this would be a dream.

[dlespiau]

Those are good arguments and a nice list of ideas :) I am personally convinced this is the right direction.

There are a few things to think about:

• should providers be in the jk repo (at least to start with), separate repos?
• what would the interface between jk and the providers look like?
• we'd want to be able to serve providers, github pages seems fine, we already host a site there. I like this a bit more than github releases so we can decide on what the URL should look like and how to store multiple versions, store manifests if needed, ...
• what would retrieving a value from a provider look like from the js side. Following the first example, we need have some idea how the k8s.helm.v2.Chart function is actually written. That will probably influence the interface needed between jk and the provider.

I'm removing the transformations as we're doing that in an orthogonal way, so:

const redis = new k8s.helm.v2.Chart("redis", {
    repo: "stable",
    chart: "redis",
    version: "3.10.0",
    values: {
        usePassword: true,
        rbac: { create: true },
    },
 });

We probably want promises in that API to enable concurrency so that may look like:

k8s.helm.v2.render("redis", {
    repo: "stable",
    chart: "redis",
    version: "3.10.0",
    values: {
        usePassword: true,
        rbac: { create: true },
    },
 }).then(objs => std.log(obj));

The Chart function may look like:

// Could be exposed to the user or locked transitively when locking the version of the library
// providing this render function (in this case it would behttps://github.com/jkcfg/kubernetes)
const providerVersion = '0.1.0';

function render(name, params) {
   return std.plugin('https//jkcfg.github.io/providers/helm', providerVersion, params).then(chart => parse YAML stream and return array of k8s objects)
}

std.plugin would be an new standard library function, a very basic interface could be:

• url: where to fetch the provider from
• version: provider version, we may want to make that part of the URL, doesn't really matter
• params: just a JSON object. The meaning of the structure of that JSON object is provider-specific
• return value: We get a return value back from the provider. This needs #148 I think, maybe we should get a Uint8array back from the provider and and the js code will have the knowledge to make something useful out of it. We should be able to reuse the mechanism in #148 to parse this data (YAML stream) and return an array of k8s objects.

A few words about how the standard library works:

• The js side and the go side exchange flatbuffers buffers. For instance this is the fb definition for the message implementing std.read
• The flatbuffer we send is really a union of all possible requests the js side can make to the go side.
• We then have a js wrapper that expose a std library function, build the fb buffer and send it to the go side. This is the std.read implementation.
• In the case of std read, but also here, it gets a bit more complicated because we have to return a promise. In our implementation, a promise is really just a serial number and we resolve the promise later when the task has finished and have a result ready.
• We end up in the std library dispatch function that routes the read fb to the right function