friction report: developing a simple generator function

[howardjohn]

This is not a bug really; rather a documentation of my experience trying to write a simple generator function. I have always found friction reports super helpful for me, but feel free to close this immediately.

Goal: write a generator that takes in a config (declaratively) with two values: revision and id, and fetch the grafana dashboard (ie https://grafana.com/api/dashboards/9614/revisions/1/download) into a ConfigMap local yaml file.

Timeline: about 3 hours

I first started digging into https://kpt.dev/book/05-developing-functions/02-developing-in-Go and the github.com/GoogleContainerTools/kpt-functions-catalog for some real examples. Right off the bat I ran into issues; the framework.Command method no longer exists in new kyaml versions, and kpt-functions-catalog uses a newer version (v0.10.21) than the docs. I don't know the context, but no deprecation period and a breaking change in a patch release seems not very user friendly.

Once I got a hello world compiling, I quickly found it challenging to debug at all to get a feel of the inputs/outputs I was expecting. Documented more in https://github.com/GoogleContainerTools/kpt/issues/2434.

Next I started looking for some examples of generators; however, almost everything in kpt-functions-catalog is just doing transformations on existing objects or validation. UpsertResource was somewhat similar but a bit complex to follow, so I was somewhat on my own. Luckily the task was trivial - I just needed to create a single configmap.

I started out just trying to do that, and not worry about getting the actual data into the configmap yet. I see I need to create an RNode and insert it into Items... should be simple enough. RNode has very somewhat minimal documentation on how to actually create one, so I dug through the ~100 methods it has and found a few promising ones: ConvertJSONToYamlNode and FromMap. I also found I could just yaml.Unmarshal into one (I think?).

I first attempt to create a v1.ConfigMap object, marshal to yaml string, then unmarshal into RNode. However, yaml.Marshal is actually outputting a completely incorrect format for these (missing type data, keys are all lowercase like resourceversion, etc), so this didn't work. I then attempted to use the stdlib json.Marshal, but ran into the fact that the type metadata is not auto populated then. I started trying to find some examples of actually doing this with schemes, decoders, etc but gave up after a few minutes, figuring this must not be the recommended approach to do this or it would be easier+documented.

From the looks of things, all examples are using yaml.Parse with Sprintf, or go templating. I would prefer strongly typed v1.ConfigMap, but was willing to compromise with go template, especially since TemplateProcessor seemed to handle a bunch of things.

Because my code wasn't strictly as straightforward as plumbing some data from input to output, I had created my own Processor that then called TemplateProcessor{}.Process(resourceList). This turned out to not be right; TemplateProcessor{}.Process only works when used directly, as TemplateData will get set to the resourceList.FunctionConfig. Makes sense after running through a debugger, but got me stuck for 10 minutes or so. Replacing Process with Filter got things rolling again, and I finally got configmap written to a file.

However, I didn't like the filename. There was no obvious method on RNode to set the filename, but I stumbled upon a config.kubernetes.io/path at some point in my debugging so tried it out and it worked; it appears nowhere in the kpt.dev documentation though.

Next, I just needed to get the grafana dashboard downloaded and into the configmap. The download part was simple enough, but then I ran into issues with my previous choice to use go templating -- because the data is json (ie multiline, has quotes, etc), we have escaping issues just sticking it into the configmap like

data:
  "{{.Name}}.json": {{.Dashboard}}

Because TemplateProcessor doesn't add in sprig functions, the usual fun tricks of helm (which I am using kpt to avoid using :slightly_smiling_face: ) like |ident 4 won't work either. There didn't seem to be a way to add functions to TemplateProcessor either.

After trying a few variations of the template, I still couldn't get the escaping to work.

My end code:

```go package main import ( "fmt" "io/ioutil" "net/http" "os" "sigs.k8s.io/kustomize/kyaml/fn/framework" "sigs.k8s.io/kustomize/kyaml/fn/framework/command" "sigs.k8s.io/kustomize/kyaml/fn/framework/parser" ) type Dashboard struct { Revision string `json:"revision"` ID string `json:"id"` } func main() { asp := GrafanaProcessor{} cmd := command.Build(&asp, command.StandaloneEnabled, false) if err := cmd.Execute(); err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } } type GrafanaProcessor struct{} func (urp *GrafanaProcessor) Process(rl *framework.ResourceList) error { c := rl.FunctionConfig data := c.GetDataMap() db := Dashboard{ Revision: data["revision"], ID: data["id"], } js, err := fetchDashboard(db) if err != nil { return fmt.Errorf("fetch dashboard: %v", err) } added, err := framework.TemplateProcessor{ TemplateData: map[string]string{ "Name": c.GetName() + "-dashboard", "Namespace": c.GetNamespace(), "Label": "grafana_dashboard", "Dashboard": fmt.Sprintf("%q", js), }, MergeResources: true, ResourceTemplates: []framework.ResourceTemplate{{ Templates: parser.TemplateStrings(` apiVersion: v1 kind: ConfigMap metadata: name: "{{.Name}}" namespace: "{{.Namespace}}" annotations: config.kubernetes.io/path: "{{.Name}}.yaml" labels: "{{ .Label }}": "true" data: "{{.Name}}.json": {{.Dashboard}} `)}}, }.Filter(rl.Items) for _, a := range added { s, _ := a.String() fmt.Fprintln(os.Stderr, s) } rl.Items = added return err } func fetchDashboard(db Dashboard) (string, error) { url := fmt.Sprintf("https://grafana.com/api/dashboards/%s/revisions/%s/download", db.ID, db.Revision) resp, err := http.Get(url) if err != nil { return "", err } defer resp.Body.Close() b, err := ioutil.ReadAll(resp.Body) if err != nil { return "", err } return string(b), nil } ```

Next I tried to backtrack, and instead use the typed ConfigMap. I realized I can use sigs.k8s.io/yaml instead of sigs.k8s.io/kustomize/kyaml/yaml and get proper marshaling; that worked. Now just needed to unmarshal it into a node

Ran into an interesting quirk of the library This panics with obscure error:

    var node *yaml.Node
    if err := yaml.Unmarshal(b, node); err != nil {
        return err
    }

This works:

    var node yaml.Node
    if err := yaml.Unmarshal(b, &node); err != nil {
        return err
    }

Once that got moving, I found I was actually wrong - sigs.k8s.io/yaml does NOT automatically work, and I got an error that my output is not a KRM format. Oops. Since my function is so simple, I guess I will bite the bullet and just manually specify the type meta...:

        TypeMeta: v1.TypeMeta{
            Kind:       "ConfigMap",
            APIVersion: "v1",
        },

Next all I needed to do was get the merging TemplateProcessor was previously giving me for free. A quick look through the code led me to the MergeFilter. This might cause issues since I really want a replace, not a merge, but should be good enough for now:

    rl.Items = append(rl.Items, yaml.NewRNode(&node))
    rl.Items, err = filters.MergeFilter{}.Filter(rl.Items)

And finally things are working!

End code:

```go package main import ( "fmt" "io/ioutil" "net/http" "os" corev1 "k8s.io/api/core/v1" v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "sigs.k8s.io/kustomize/kyaml/fn/framework" "sigs.k8s.io/kustomize/kyaml/fn/framework/command" "sigs.k8s.io/kustomize/kyaml/kio/filters" "sigs.k8s.io/kustomize/kyaml/yaml" kyaml "sigs.k8s.io/yaml" ) type Dashboard struct { Revision string `json:"revision"` ID string `json:"id"` } func main() { asp := GrafanaProcessor{} cmd := command.Build(&asp, command.StandaloneEnabled, false) if err := cmd.Execute(); err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } } type GrafanaProcessor struct{} func (urp *GrafanaProcessor) Process(rl *framework.ResourceList) error { c := rl.FunctionConfig data := c.GetDataMap() db := Dashboard{ Revision: data["revision"], ID: data["id"], } js, err := fetchDashboard(db) if err != nil { return fmt.Errorf("fetch dashboard: %v", err) } cm := &corev1.ConfigMap{ TypeMeta: v1.TypeMeta{ Kind: "ConfigMap", APIVersion: "v1", }, ObjectMeta: v1.ObjectMeta{ Name: c.GetName() + "-dashboard", Namespace: c.GetNamespace(), Annotations: map[string]string{ "config.kubernetes.io/path": c.GetName() + "-dashboard.yaml", }, Labels: map[string]string{ "grafana_dashboard": "true", }, }, Data: map[string]string{c.GetName() + ".json": js}, } b, err := kyaml.Marshal(cm) if err != nil { return err } var node yaml.Node if err := yaml.Unmarshal(b, &node); err != nil { return err } rl.Items = append(rl.Items, yaml.NewRNode(&node)) rl.Items, err = filters.MergeFilter{}.Filter(rl.Items) if err != nil { return err } return nil } func fetchDashboard(db Dashboard) (string, error) { url := fmt.Sprintf("https://grafana.com/api/dashboards/%s/revisions/%s/download", db.ID, db.Revision) resp, err := http.Get(url) if err != nil { return "", err } defer resp.Body.Close() b, err := ioutil.ReadAll(resp.Body) if err != nil { return "", err } return string(b), nil } ```

Overall, there was no one thing that was broken/bad/hard - but rather at most steps I encountered small barriers that overtime led to a more frustrating experience than I expected. I hope over time as the ecosystem matures a bit this will get streamlined a bit - I think a large function catalog would be great for the project

[mengqiy]

Thanks for this detailed friction log! We will address them.

[mengqiy]

Some of the low-hanging fruits has been addressed in Q3. We are working on improving the golang SDK in Q4.

[bgrant0607]

See also #2528