add core (sprig) functions to text/template

[kghenderson]

Is your feature request related to a problem? Please describe. Not an existing problem, but is a constraint and affects current, existing projects and cue adoption.

Propose adding: https://github.com/go-task/slim-sprig functions to core cue text/template library. Note that slim-sprig is based on https://github.com/Masterminds/sprig but removes the slow performing ones.

First, I believe that CUE should differ from Go here and that's fine, because CUE should be, imho, a tool and language installed on ordinary business users machines and should not require downloading anything externally. Imported packages are necessary and great and amazingly useful but are more for application/service developers than ordinary power-users.

CUE should be the swiss army knife for formats and the cue text/templates should be easily written and shareable amongst these types of users. Also, they typically won't have control over their source files or really know how to re-transform them beyond the equivalent of a curl/wget/openapi retrieval ... ideally they should be able to download a json file, which everybody has consistently the same, and write a simple template transform to convert it into other formats. Particularly formats which are in-house or custom which require special processing, file splitting, etc.

A recent case in my own work is to text/template to an in-house developed hjson file. I first tried to use an hjson-cli but it automatically sorts the keys and doesn't trim empty lines. Arguably this is "right" and technically sound, to a computer there's really no difference.. but here's the rub ...

If I pull and write to my code repository with Cue instead of the other in-house developed tool, then nearly every single file shows as being changed in Git and we're relying on Git to handle detecting those changes automatically based on the file contents. I'm sure some people would argue (myself included) that sorted keys are more stable and deterministic and because it's a map you're not guaranteed order anyway, so it shouldn't matter and you should fix the other tool.... i understand, that is right ... however ... it doesn't matter, i cannot change that other tool - never going to happen, and this is corporate life.

Not having these custom format transform options in cue means that without very extensive and cumbersome cue code to transform the source before passing to templates ... means that I can't easily make a drop in replacement, cue won't play nice with the ecosystem and so practically speaking won't be used for that purpose.

an example of a simple, but very major hurdle here is simply indenting a multiline cue string into an indented block in my output file format.

Source Value in CUE

CueValue: """
  This works great
  No problems, multi-line string
    - mixes of tabs for cue-indent, but spaces within this text block is confusing and awkward
    - both make the editor show tabs differently and we're ok
    - no problem here 
  """

here's want i need:

YamlThing:
  Boilerplate
    MoreBoilerplate:
       block needs to go here:  |2-
          CueValue goes here
          With all lines in cue string
          just indented. 

to do that today: i had to completely make a new struct with indent fields (because i need to pass this new thing into the template), manually deep copy every single field from the base struct to my indent-handling one (i wish there were an easier way to do this too), then ... i had to wrestle with some very fun regular expressions (which I rely on other tools to help me with - future documentation topic) ... to find any newline and replace with newlines with indent spaces, then after that go back and trip any lines with newlines and only spaces. to trim the empty lines. good times. not saying this is good, but i must match what they have in git for better or worse.

so this works, but it ain't simple, and it's what templates are already designed to deal with. go is for developers and packages are great. cue is for developers but also casual business users (for data validations), and IT departments rightfully control those environments, and this major use case should be ideally be covered by default.

Describe the solution you'd like

YamlThing:
  Boilerplate
    MoreBoilerplate:
       block needs to go here:  |8-
          {{ indent 8 CueValue }}

^ way easier to write, way easier to read, very simple for casual users https://go-task.github.io/slim-sprig/strings.html#indent

Describe alternatives you've considered As described above, this requires doing the work in cue to make what the template needs first.

I considered using let statements in a loop and doing the transform there and passing additional arguments to the template, but given that currently cue can't export tools and the templates are very fragile, you're essentially working blind.

So by making a new struct and keeping that in a new package, but a non-tool file, and working in that realm, I was at least able to see what I was doing as I worked.

Additional context Json->Template transforms, using cue without writing cue, can eventually be streamlined too, but easy to copy/paste from an existing example.

Note that I do generally doing work in Go/Cue and not text/templates where possible, still a good practice for devs.

[kghenderson]

additional note that cue templates and go templates, given the same source structs, should ideally be interchangeable, i.e. that cue functions and go functions could be called from the template the same way. that said, the implementations behind those functions can of course be different.

[kghenderson]

adding a note that sprig & slim-sprig both also support reflection functions: http://masterminds.github.io/sprig/reflection.html which could also possibly serve as a workaround to type reflection from cue

[myitcv]

In general I don't think we're against adding clearly useful, well-defined functions to the default function map for text/template. Could we perhaps make the proposal concrete by taking a couple from the list? Would indent be your first suggestion?

Stepping back, I wonder whether the package management, functions and WASM work from @rogpeppe and @4ad could come into play here. Combining the two, would it be possible to actually pass a function map to text/template.Execute from CUE? If so, that would allow us to see what functions are generally useful, and provide a potential transition path to the core text/template package.

First, I believe that CUE should differ from Go here and that's fine, because CUE should be, imho, a tool and language installed on ordinary business users machines and should not require downloading anything externally

The one slight exception that I personally would apply here is that it's sometimes hard to be sure whether something belongs in core CUE or not. Hence the reference to the three linked issues above. If we can experiment in non-core packages, that helps us understand whether something should be core or not. Right now we would be limited to marking parts of the pkg/... API surface as experimental... which doesn't feel ideal.

To be clear, if a use depended on a non-core package, with a full modules implementation the use of that external dependency would be seamless (just as a go build seamlessly fetches dependencies).

But in the short term, per my suggestion above, I suggest we make this concrete with a couple of functions.

If I pull and write to my code repository with Cue instead of the other in-house developed tool, then nearly every single file shows as being changed in Git and we're relying on Git to handle detecting those changes automatically based on the file contents. I'm sure some people would argue (myself included) that sorted keys are more stable and deterministic and because it's a map you're not guaranteed order anyway, so it shouldn't matter and you should fix the other tool.... i understand, that is right ... however ... it doesn't matter, i cannot change that other tool - never going to happen, and this is corporate life.

I got slightly lost here. Is the issue you are alluding to that text/template sorts the argument to range?

# Export data to see order
exec cue export --out cue -e data
cmp stdout stdout.golden

# Run template to see order
exec cue export -e output --out text
cmp stdout stdout.golden

-- x.cue --
package x

import (
    "text/template"
)

data: {
    b: 1
    a: 2
}

tmpl: """
    {{ range $key, $value := . -}}
        {{ $key }}: {{ $value }}
    {{ end -}}
    """

output: template.Execute(tmpl, data)
-- stdout.golden --
b: 1
a: 2

This currently fails with:

# Export data to see order (0.021s)
# Run template to see order (0.017s)
> exec cue export -e output --out text
[stdout]
a: 2
b: 1

> cmp stdout stdout.golden
--- stdout
+++ stdout.golden
@@ -1,3 +1,2 @@
-a: 2
 b: 1
-
+a: 2

FAIL: /tmp/testscript2906628232/repro.txtar/script.txtar:7: stdout and stdout.golden differ

(note sure where the extra blank line is coming in).

If this is the issue, I'd suggest we file a separate issue for that. It's close related to #474.

so this works, but it ain't simple, and it's what templates are already designed to deal with.

Completely agree. Let's make this concrete with indent and whatever other functions you would suggest.

[kghenderson]

thank you for the reply :-) i'll try to pick apart the questions, but please let me know if i've missed something.

re: "core" library, i was using a rather loose interpretation simply meaning: included with cue distribution, so if this were to be accepted, then a sub-package which can be imported the same as list or strconv, or inside of text/template is perfectly acceptable. i agree there can be a starting point for just the basics one might expect for building a template.

the request for common template functions, e.g. indent, is orthogonal to the request for access to the function map when executing a template from within cue. both are desirable for opposite ends of the spectrum:

at the easy end, i could provide a json file (or really any cue-supported format), and simply pass it to template, and be able to do a basic transform without really needing to do anything special. slim-sprig: https://github.com/go-task/slim-sprig#principles-driving-our-function-selection is itself already an effort to extract the most common and dependency-free functions from the fuller (non-slim) sprig library. goal is to have a regular office worker be able to take an extract (e.g. a graphql query provided to them), then easily turn that into a markdownish "report" or something which can subsequently be piped into a pdf for distibution.

at the complex end, more advanced transforms will require 'developer' support and then we'll want to pass custom functions into the template, e.g. for the use in pipelines, but we also might just use pure cue to do the transform prior to it reaching the template, perhaps an example is to merge in environmental settings/attributes/tags into the template as well.

re: the rant on the custom hjson report, it's a bit off topic, except that ... the template requests make seem nit-picky (even to me!), but this feature request (someday) and those nit-picky adjustments could be the difference between being able to work for the particular job or not. (in this case because even if the whitespace indentation is slightly off it means it can't be used as a drop-in replacement for the other tool - i.e. and impediment to implementing cue in the workplace)

i'll also point to this interesting utility: https://pbar1.github.io/gq/ and https://9to5tutorial.com/go-template-the-best-programming-language which include sprig functions: point being these are so common and necessary as they probably should get accepted (someday), even into go's default template functions.

[kghenderson]

lol, i love that there's also a: https://github.com/hairyhenderson/gomplate and before you ask: no, there's no relation :-)