Inconsistent use of the terms 'annotation' and 'directive' in docs

[gcheadle-vmware]

Acceptance Criteria

As a ytt user looking at the "When should I include a space in my ytt comment?" question in the faq I want to see an answer explaining why some annotations use spaces and some don't.

As a ytt user looking at the docs I want to search the docs for "spaces in annotations", "#@ overlay or #@overlay", or "When do include a space in my annotations?", and see a result pointing to a FAQ response answering the question.

As a ytt user looking at the docs I want to search the docs for "templating directive", or "directive", and see a result pointing to Text Templating.

Additional doc fix: In the Known Limitations it mentions a "templating directive", I think these references can be changed to "annotation".

---

Original issue context:

It was pointed out on slack that in the faq defines a clear boundary between annotations and directives. This definition is confusing and it contradicts how we define a yaml annotation in the yaml primer doc.

@ewrenn8 noted: "All #@'s are annotations, the #@ 1+1 is just syntactic sugar for the full annotation #@temeplate/code 1+1"

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible" 👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[GrahamDumpleton]

A further specific point of confusion is that the docs state:

Annotations do not have a space, and they refer to a given node in the tree

Specifically, "they refer to a given node in the tree".

When including Starlark code using #@, ie., with a space, it need not refer to a node and so is wrong to say it refers to a node.

For example, a variable assignment at the top of a file.

#@ variable = "somevalue"

So if an annotation is defined as referring to a given node, it is wrong to say that all #@ comments are annotations.

I think moving to the idea that everything is an annotation is wrong. At the same time I think the terms annotation and directive are possibly too close in meaning to refer to the different cases.

What is currently referred to as directives, with the space, is Starlark code, and for understandability I think it is better to be quite clear that stuff with a space is code. It makes more sense to see data.values.variable as code which is using dotted notation to look up a variable on the data object imported using the load statement, than some magic string which is an annotation.

I'll probably have more to say on the language used to explain things once done with some workshops which cover ytt.

[GrahamDumpleton]

Looking into this more, one can notionally say a subset of what is Starlark code, ie., with space, can be an annotation against a node. In this specific case you are restricted to the Starlark code being an expression, with the result of the expression being applied to the specific node it is applied against. In this case the annotation is always placed in the position of the value for the node, and not on the line before as with overlay annotations.

So this in effect is a third type of usage distinct from overlays (which apply to a map item), and general embedded Starlark code statements.

There is also a fourth distinct usage, which is #@data/values, which is applied to a YAML fragment starting with ---, although it could also be said to apply to the file as a whole, since you can't have more than one YAML fragment in a data values file. The #@data/values-schema falls in this same category as #@data/values.

So there definitely are subtleties and it isn't as straight forward as annotations vs directives.

[GrahamDumpleton]

One reason for not using the term directive is that the YAML specification has a concept of directive.

• https://yaml.org/spec/1.2/spec.html#directive//

This states:

Directives are instructions to the YAML processor. This specification defines two directives, “YAML” and “TAG”, and reserves all other directives for future use. There is no way to define private directives. This is intentional. Directives are a presentation detail and must not be used to convey content information.

It may therefore be wise to not use the term to mean something different in ytt to avoid confusion.

[GrahamDumpleton]

Summarising how I see things right now, we have the following.

Annotation Types

There isn't just one type of annotation, but really multiple. I try and split these up in reference to how they are applied to the YAML and not a higher level purpose.

Structure Annotations

Examples of these are #@data/values and#@data/values-schema.

These are applied to a YAML structure starting with a line ---.

For example:

#@data/values
---
a: 1
b: 2
c:
  d: 4

Node (object) Annotations

Examples of these are overlay annotations #@overlay/merge, #@schema/type and #@schema/nullable.

These are applied before a YAML node (object) within a YAML structure. This can be an object, an object property (map item), or a list item.

For example #@schema/nullable in:

#@data/values-schema
---
#@schema/nullable
aws:
  username: admin
  password: "1234"

Annotations of this type related to overlays are more specifically called "Action Annotations", thus overloading the annotations term even further.

Value Annotations

This relates specifically to Starlark code expressions of the form #@ expression, with space after #@.

These are applied in the context of where a value is being supplied for an object property (map item) or list item.

For example:

---
a: #@ expression

The YAML value in these cases is left unset and only the value annotation is added.

Code Statements

These are arbitrary code statements that can appear anywhere in the YAML file. In the case of statements which can enclose a block, eg., def, if and for, they can enclose YAML structures or objects, as well as other code statements.

All code statements are of the form #@ statement. Code statements that can enclose a block are ended using #@ end.

Expressions within value annotations can refer to variables set by prior code statements.

For example:

array:
#@ arr = [1,5,{"key":"val"}]
#@ for i in range(len(arr)):
- val: #@ arr[i]
  index: #@ i
#@ end

Overlay Matching

I have split out overlay matching as being separate. Here I am referring to #@overlay/match and #@overlay/match-child-defaults and which documentation calls "Matching Annotations".

These are different because they can be applied as both structure annotations and node (object) annotations.

For example:

#! selects over the list of documents
#@overlay/match by=overlay.all, expects="1+"
---
items:
#! within *each* matched document,
#!   selects over the array in `items`
#@overlay/match by="id"
- id: 2
  modified: true

Conclusions

The current usage of the term annotation is confusing.

It is used for the three different types of annotations described above when applied to YAML.

At the same time, within documentation of overlays it refers to "Matching Annotations" and "Action Annotations".

My suggestion therefore is that the concept of annotation be restricted to the concept of applying a comment to YAML, with the three variants above.

The overlay documentation should not use the term annotations in "Matching Annotations" and "Actions Annotations" and find a different way of describing those two categories of how the annotations are used.

Where Starlark code is used, you have value expressions which are an annotation, but also general embedded code statements which are not. This is despite that in both cases a space is required after #@.

The term directive should not be used at all to avoid confusion with the past. Personally to me if defining this at the outset, I would have used "Matching Directives" and "Action Directives" to distinguish the different categories of overlay annotations, but is too late for that now. The term directives never made any sense to me as meaning anything with a space after #@, especially as it also seemed to be used in a more general way in saying template directives.

cc @jorgemoralespou

[aaronshurley]

@GrahamDumpleton This is all really great elaboration and feedback. Thank you!

I'm going to move this item to carvel accepted as we can certainly improve the docs here. Exactly how we do that might be tough without actual changes (i.e. PRs) to review, but this is great fodder for that next step.

This likely won't be prioritized immediately but we'll likely have some dedicated docs time in our near-term future. If anyone is feeling ambitious, please feel free to contribute a PR to get this ball rolling.