Support Jsonnet --[tla|ext]-[code|str]-file

[peter-mogensen]

When working with inline environments in order to not commit environment specific data to the general tanka package git repo it's really useful to not only be able to inject apiServer and namespace, but also any config overrides in form of a larger JSON config object. If there's a lot of config this can get tedious to supply with --tla-str ... also, any --ext-code has to exist in order to not raise an error, so it doesn't work for having default values. Encoding long JSON object directly in the cmd-line is also ugly and hard to debug.

It would be really nice if Tanka supported Jsonnet's --tla-str-file/--tla-code-file/--ext-str-file/--ext-code-file arguments.

Unfortunately it seems both #168 and #353 got closed without addressing this issue.

[Duologic]

Regardless of implementing this feature or not, it is a good (gitops) practice to commit 'environment specific data' somewhere at least. Perhaps you can further explain your use case so it becomes apparent why commiting certain data somewhere is not useful.

More generally, there is this blog post about running Tanka (inline) environments at scale, this is how we scaled from a few hundred to ~2500 Tanka environments at Grafana Labs today.

[peter-mogensen]

Sure ...

First, this is not a gitops enviroment. We are not having the clusters automatically monitor and sync with a git repo. We're deploying with terraform and tanka.

That said ... of course environment configuration is committed to git. (Just not the git repo with the tanka package) Especially the main shared environments. But we also have ephemeral personal environments for development and we don't want to include that configuration in merge-requests, so it's not committed along with code that needs to migrate towards production branches. Such local override of default environment configuration is kept separately for where it's needed.

[peter-mogensen]

Adding those cli options in the same style as the jsonnet tool turned out to be a very simple patch.

Here's a proposal using the same option parsing as the rest of the --tla/ext options for the tk tool implemented the exact same way as the jsonnet cli tool implement the "-file" options - by just passing a "import" expression on to the jsonnet engine.

https://github.com/peter-mogensen/tanka/commit/fad3cc6f9f59b3178e6d90f50ad1bac716c5ad5a

This allows you to import top level data from file without having jsonnet complaining about "computed import paths".

I really see no reason not to support this - but all the good reasons everybody else has given in the 2 issues referenced above.

[sigwinch28]

I have used @Duologic 's suggestion from https://github.com/grafana/tanka/discussions/608#discussioncomment-1434373 to help me migrate from an external tool to tanka. The key is that you can accept jsonnet code on stdin as either a TLA or extVar.

I generate a data structure which resembles the output of the helm and kustomize utils in tanka and pass that on stdin to tanka.

Here's the MWE bash script I use to wrap tk:

#!/usr/bin/env bash
set -euo pipefail
COMMAND="$1"
ENVIRONMENT="$2"
shift 2
my-fancy-generator-thingy | tk "${COMMAND}" --ext-code "myFancyGeneratorThingy=import '/dev/stdin'" "${ENVIRONMENT}" "$@"

then I use a jsonnet library to abstract over and access this data structure:

// lib/my-fancy-generator-thingy.libsonnet
local configs = std.extVar('myFancyGeneratorThingy');
{
  get:: function(configName)
    std.get(configs, configName, error 'fancy thingy ' + configName + ' not available. Did you follow the README?'),
}

// environments/yeet/main.jsonnet
local mfgt = import 'my-fancy-generator-thingy.libsonnet';
{
  something: mfgt.get('key-goes-here')
}

I too see no reason not to support the upstream args, but I think there's an immediate workaround either by doing something like @Duologic has previously suggested, or generating --tla-code args on the fly.

[peter-mogensen]

While I agree that that's a nifty workaround, I actually strongly would prefer --tla-code-file ... even if that means I have to carry a patch for tk.

The reason being that we have terraform invoke tanka and changing the command line to wrap tk in "fancy-generators" or tools like sops (as suggested in one of the linked issues) quickly create loss of generality.

In essence... we basically use terraform the same way as one would have done with the helm-provider, but use tanka instead of helm because that's the better tool.

[peter-mogensen]

While I still think there's no reason not to support this (and it's an easy fix), we ended up concluding that in the end our use case was better suited by writing a tanka provider for terraform. (so we got a better "destroy" experience)

[zerok]

Hi :) Sorry for the long delay. I've now moved this ticket to our internal backlog. No timeline but it's progress ^_^

[rudo-thomas]

I implemented this in https://github.com/grafana/tanka/pull/1238 :)