Proposal: Nested variables in template

[forrestli74]

Please label this issue as fit. Intentionally avoided any terminology around base16, as I'm still very confused about it.

This is yet another idea brought from base9.

We should use nested variable names (i.e. part1.part2) instead of - separated.

This has two aspects to it. The second builds on top of the first one, but both can be considered independently.

formatting

Use base03.hex instead of base03-hex

Advantage:

Instead of

<string>{{base07-dec-r}} {{base07-dec-g}} {{base07-dec-b}} 1</string>

we can write

{{#base07}}<string>{{dec-r}} {{dec-g}} {{dec-b}} 1</string>{{/base07}}

Which is much easier for template maintainer.

For more complicated templates, we could potentially use mustache partials

{{#base07}}{{>partial}}{{/base07}}

semantic aliasing

when defining semantic aliasing instead of

button.foreground: "#{{button_foreground.hex}}"
button.background: "#{{button_background.hex}}"
terminal.red: "#{{ansi_red.hex}}"
terminal.green: "#{{ansi_green.hex}}"

we can say

{{#button}}
button.foreground: "#{{foreground.hex}}"
button.background: "#{{background.hex}}"
{{/button}}
{{#ansi}}
terminal.red: "#{{red.hex}}"
terminal.green: "#{{green.hex}}"
{{/ansi}}

For semantic alias override:

instead of

button_background: base09
button_foreground: base00

we can say

primary_normal:
  bg: base00
  fg: base09
primary_inverted:
  bg: base09
  fg: base00
primary_button: primary_inverted

or

button:
  bg: base09
  fg: base00

You get the idea. Which style/syntax/alias we support in the spec can be debated more in detail in follow up discussion, but I just want to introduce this idea first discuss why nested scopes are good.

Advantage:

1. easier to read the spec/aliases
2. easier for scheme developer to override the default if we decide to support something like in the primary_inverted example.
3. we can group domain specific semantics together for easier template development. (i.e. ansi, markdown)

All these are already implemented in base9, and in my experience, it works pretty well. base9 alias config: https://github.com/base9-theme/base9-builder/blob/main/src/default_config.yml#L19 base9 template that uses nested variables heavily: https://github.com/base9-theme/base9-builder/blob/main/templates/preview.mustache

[joshgoebel]

Ok, you're talking about a lot of different things here, so one at a time:

• Use . vs - for nicer readability; I'm largely on board with this idea overall, but I think it's a bit of a builder change to support both ways... 😕
• Partials. Don't see the need for most templates. (I feel like we'd need our own micro format to declare them all in a single file, etc)
• I feel like separate fg/bg is a whole other complex discussion apart from how we'd encode it in YAML. (I've mentioned this in other places)
• Tiered scopes and YAML, why not go all the way with it:

term:
  ansi:
    red: '#ff0000'
ui:
  background:
    bright: "#fffffff"

# vs
"term.ansi.red": '#ff0000'
"ui.background.bright": "#ffffff"

Personally I see these things as things we could consider after the big feature push on Base17 itself done... these to me are more about "niceness" and "refinement"... and the first step is creating our own spec (not Base16) that we're free to push forward and iterate on as it's own thing.

@belak Is it easy to drop in a custom lookup resolver in Go/YAML? In JS it's trivial to pass in a context object to the renderer that provides it's own lookup method so I can so as much (or as little) as I want when I'm given the name of something to lookup (the entire template variable resolution could be virtual, etc)... so switching to . or supporting both is well within grasp. I dunno if Go is less flexible?

[forrestli74]

Partials

Regarding partials, I agree it's not needed for most templates. But a few templates could really benefit from it. (GitHub search something like base05-dec-r would give you some examples)

Partials probably shouldn't be supported right away. But I just saying it opens up opportunity for that, which is a point to switch to dotted version.

Nested scopes

Tiered scopes and YAML, why not go all the way with it:

I'm not sure if you are been sarcastic. But there is a balance of how much nesting we have. And can be discussed separately. If the first iteration of base17 does not have any concept of nesting to keep things simple, then so be it. But if we do have variables like ansi_red, we should support it through dotted format.

custom lookup resolver

Are you talking about partials or just dotted variables?

For dotted variables, as long as you generate nested object, things like {{#c1}}{{hex}}{{#c1}} is supported in most of not all languages, since it's in mustache spec. And this is pretty easy to implement. If you don't want to generate all formats up front, you could use getter function in JavaScript, but you are out of luck for other languages, since they don't have anything like that.

For partials, only JavaScript supports custom lookup. Must other languages only support looking up relative to the template file location.

I'm pretty sure Go falls into the "Other language" category in both cases. Belak feel free to confirm for golang.

[joshgoebel]

I'm not sure if you are been sarcastic.

I wasn't at all, I personally don't mind the above examples... 5 or 6 might start to get a little silly though, but most TM scopes top out at 3-4 levels anyways... It feels (at a glance) a bit more readable than tons of "x.y.z.b" keys... I do feel like I'd really want to write out a full theme in practice though to make a final decision. But as a said this might be more for "stage two". And it shouldn't be hard to support both formats if we so desired.

For dotted variables, as long as you generate nested object,

Yes, I'm aware that's one route...but at least in JavaScript Mustache it's trivial to get the entire name "x.y.z.booger" and then calculate the return value however you desire... I plan to use this in Node builder in the future to avoid building an HUGE object with 50,000 keys when generally only 250 of them might ever be used by any given template. This makes a lot more sense for the template to "render" the desired output rather than the schema calculating a zillion output keys. That coupling is far too tight - the schema should not know so much about the duties of the template.

• schema provides the resolver (turning names [baseXX, variables, semantics] into hex colors) since it knows all the semantica and all the hex colors
• template defers to schema resolver to get hex colors, then calls any specified transform function (.hex-dec-r) on that color, etc...

but you are out of luck for other languages, since they don't have anything like that.

Many dynamic languages do, Ruby just to name one... I wouldn't be surprised if other Mustache libraries also let you substitute your own Context object that is capable of doing it's own resolving when given a fully qualifier template variable name. So you don't need language support if you have good library support.

[belak]

but you are out of luck for other languages, since they don't have anything like that.

Many dynamic languages do, Ruby just to name one... I wouldn't be surprised if other Mustache libraries also let you substitute your own Context object that is capable of doing it's own resolving when given a fully qualifier template variable name. So you don't need language support if you have good library support.

Just as a note, all the Go and Rust mustache libraries I've looked at don't have this.

[forrestli74]

Cool, cool. I don't have anything to add. Feel free to close this issue, or somehow archive it and revisit later.

[joshgoebel]

Go has lookup broken out pretty well: https://github.com/hoisie/mustache/blob/master/mustache.go#L346

And it's only called in two places... so it seems possible that this feature could potentially be added to Go - replacing the lookup with a callback function that you'd perhaps need to specify on a template's struct - or pass to the top level render function? Probably taking the same type of arguments as the existing lookup... and then it would also be free to generate data "on the fly" rather than being bound to static data lookups. I'm spitballing a bit since I don't know Go super well but I'd be pretty disappointed to learn there was no way to allow for custom callbacks - since that's a super common pattern used in so much software.

But yes, as it currently stands this approach (calc template vars JIT as needed) would seem harder to implement in Go than JS -because the library lacks built-in support... it seems you're left pre-calculating every possible value for every possible variation for every possible semantic slot - even if only 16 of them end up ever being referenced. That may not be slow in the absolute sense per se, but it does sound inefficient. 😕

This probably also makes dynamic scopes hard/impossible as well, since you'd have to precompute the entire tree in Go (and you might not even know the entire tree) where-as in JS I can just start with x.y.z.a and then walk up to x.y.z, then x.y, then x until I find a match. That is going to break the dynamic nature of scopes that I was hoping to rely on. ☹️

IE, a template should (in a perfect world) be able to specify detailed scopes like "constant.numeric.integer.hexadecimal"... and that would resolve to (in order):

• constant.numeric.integer.hexadecimal
• constant.numeric.integer
• constant.numeric
• or constant

This allows building very nuanced templates while also allowing for very thin schemas since theme designers can define as much or as little detail as they want.

[forrestli74]

@joshgoebel Those points to me sound more regarding the dynamic tree implementation, instead of points against or for the proposal itself. (Just pointing out my understanding)

So regarding dynamic vs static tree, I have a few thoughts:

1. Static trees is a perfectly ok implementation and will not be much harder to implement than no nesting. Dynamic tree is just an implementation improvement.
2. Assuming we have the same set of variables, static trees will not perform much slower than no nesting.
3. Even for static trees or no nesting, implementing something like constant.numeric.integer.hexadecimal will not be too slow for modern computers. (~10 formats ~10000 variables 100 bytes or cycles = 10 million. That's the order of magnitude for run time and memory consumption, each number is chosen conservatively)
4. Static tree is also useful for template developers, as they'd like to know exactly what mustache variables they can use.

So there is no reason to develop a dynamic tree unless:

1. We really have a huge number of variables, or
2. We want to support it on really constrained systems, or
3. You just want to build interesting stuff.

[joshgoebel]

instead of points against or for the proposal itself.

This true. I'm doing what I've gotten onto others for doing in the past. Sorry 🤦‍♂️ I just always try to write performant software when I can... so if I can do it in 10,000 cycles, I'll choose that - plus the benefit of not needing to embed/maintain a huge static list of scopes in the builder. But there is no rule saying every builder must implement the system in exactly the same way. More dynamic languages could use a dynamic implementation and more static languages could use a static implementation. Same result effectively.

they'd like to know exactly what mustache variables they can use.

It's also possible that the template publishes the scopes/color as a list... so then a template could output scopes that it knew nothing about - just so long as the receiving application understood the scope. Many editors that use raw TM scopes could take advantage of this... but now I'm probably talking more about interop and advanced use cases than the daily driver scenario. :)