Proposal: Semantic labelling markup in Yarn

Is your feature request related to a problem? Please describe.

Currently, lines delivered to the game by Yarn Spinner are plain text. When developers want to add additional formatting information to their text, like bold fonts or varying text sizes, the reply we've generally given is "implement your own markup and parse it yourself".

This approach to creating ad-hoc solutions has been a part of Yarn Spinner since the start of its history. Night in the Woods, the game Yarn Spinner was originally built for, uses two different styles of markup to annotate its lines - a BBCode style [b][/b] approach, and a second style with curly braces. Others have made use of an HTML-like markup, which has the happy side-effect of automatically working with TextMeshPro's markup in Unity.

However, as we continue to add syntax to the language that affects the contents of lines, the lack of an official markup format has lead to new features breaking users' bespoke solutions. Additionally, when users implement their own markup, they also have to take on the responsibility of parsing that markup. Given that the value proposition of Yarn Spinner is "we'll handle the parsing for you", this is less than ideal.

In short: Yarn Spinner should define its own markup language, handle the parsing itself, and provide a good API for consumers of it. This proposal defines a flexible, powerful and entirely optional markup language for Yarn Spinner.

Please note: This proposal is for the language specification only. It makes recommendations for how engine implementations should use it, but this is not a proposal for a feature in Yarn Spinner for Unity.

Describe the solution you'd like

Markup Format

The core concept of this markup is that regions of a line or option may have one or more attributes applied to them. Ranges of text in a line or option that the writer wishes to annotate are wrapped in pairs of HTML-like tags:

Why, <wave>hello</wave> there!

In this example, the word hello has the attribute "wave", while the word there has no attributes.

Attribute properties

Tags can carry additional data:

Why, <wave size=5>hello</wave> there!

In this example, the attribute has a property size, which is equal to the number 5.

The following syntax is permitted for property values:

Integers
Floating-point numbers
Strings (delimited by double quotes)
- Strings may contain escaped quotes with a backslash.
The words true or false
Runs of alphanumeric characters, up to but not including a whitespace or the end of a tag; these are interpreted as a string (e.g. <mood=happy> is interpreted the same as <mood="happy">
Expressions (delimited by curly braces), which are processed as inline expressions

Attributes can have an unlimited number of properties.

Shorthand attribute properties

Often, an attribute makes no sense without an associated property; a fontSize attribute would make no sense without a property that stores the actual size that the writer wants. To make it easier to write, a syntactic sugar is available that lets the writer implicitly create a property with the same name as the attribute, so that the following two lines are semantically identical:

Why, <wave=5>hello</wave> there!
Why, <wave wave=5>hello</wave> there!

To simplify the parser, if this syntax is used, only a single property is allowed. If you want to create two properties, you need to write them out in full, along with the attribute name.

Overlapping attributes

A range of characters is allowed to have multiple overlapping attributes. In the following example, the characters Hel have the attribute "wave", while the characters lo have both "wave" and "size".

Why, <wave=5>Hel<size=2>lo</size></wave> there!

Unlike in HTML, attributes may be closed in any order. For example, the following is not a syntax error (and is semantically identical to the previous line):

Why, <wave=5>Hel<size=2>lo</wave></size> there!

The sequence </> closes all currently open tags.

All unclosed attributes are automatically closed when the end of a line's content is reached. (In Yarn, a line's content ends at the line break, a hashtag, or a condition.) This means it is the following examples are all semantically identical:

Why, <wave=5>Hel<size=2>lo there!</wave></size>
Why, <wave=5>Hel<size=2>lo there!</size>
Why, <wave=5>Hel<size=2>lo there!
Why, <wave=5>Hel<size=2>lo there!</>

Markers

In addition to marking ranges of text as having attributes, specific points of a text may have an attribute. For example, a line might trigger a sound effect or a screen shake halfway through being delivered.

This is represented in the same way as in HTML - with a "self-closing" tag. For example:

Hello, I-- <screenshake/> what was that?!

Self-closing tags do not apply any attributes to their text, and may have properties, just like other tags.

Implicit attributes

A commonly requested feature is the ability to identify and extract the name of a character from a line.

To support this, the start of the line up to and including the first colon followed by optional whitespace, is implicitly marked with the character attribute. This attribute has a single property, "name", which is a string containing the text up to (but not including) the colon and optional whitespace. If a colon does not appear in the line, no implicit attributes are added.

This means that the following line:

Mae: Wow!

is conceptually the same as this:

<character name="Mae">Mae: </character>Wow!

Whitespace Between Tags

To increase readability when writing multiple adjacent tags, writers might add whitespace between tags. For example:

Whoa! <soundeffect=shake.wav/> <camerashake=5/> An earthquake!

Simple removal of the tags would result in three spaces between "Whoa!" and "An earthquake!", which may not be desired. To avoid this, if a tag has whitespace before and after it, all whitespace after it is removed. For example, the final plain text of this line would have a single space between "Whoa!" and "An earthquake!".

This feature is optional; dialogue runners may choose to not perform it, depending on the needs of the developer.

Escaped Characters

The < and > characters may be used in the line text, and intended to be presented to the player. These characters may be escaped using a backslash:

Here's me hugging you: \<o\>. // will not attempt to parse this as the tag <o>

Compiler Implementation

Markup parsing is not done in the compiler, and no changes to the bytecode output are needed.

Instead, this parsing is done in the dialogue runner, at the same stage where inline expressions and format functions are parsed. This allows attributes to be locale-specific, and preserved in the generated string table. This allows translators to modify and rearrange attributes as required for different languages.

Attribute property values may be constants (numbers, booleans, strings, or null), variables, or expressions. If they are expressions, the same syntax for inline expressions is used (i.e. wrapping the expression in { and }).

When a dialogue runner parses a line containing markup, it produces two outputs:

The text of the line, with all tags removed (the "plain text")
An ordered collection of attribute ranges, each containing the following information
- The index in the plain text where the attribute begins
- The length of the run of text.
- For each property of the attribute:
  - The name of the property
  - The value of the property

The collection of ranges is ordered by the start point. If two attributes begin at the same point, they are ordered based on their order in the text. For self-closed marker tags, the length of the range will be zero.

For example:

Mae: Why, <expression="grin"/> <wave waviness=5>hel<size=2>lo there!

Would result in the following output:

Plain text: Mae: Why, hello there!
Attributes
- Attribute:
  - Start: 0
  - Length: 5
  - Name: character
  - Properties:
    - name = Mae
- Attribute:
  - Start: 10
  - Length: 0
  - Name: expression
  - Properties:
  - expression = grin
- Attribute:
  - Start: 10
  - Length: 12
  - Name: wave
  - Properties:
    - waviness = 5
- Attribute:
  - Start: 13
  - Length: 9
  - Name: size
  - Properties:
    - size = 2

A dialogue runner may elect to not parse the line for markup. This means that this entire feature is optional, and at the discretion of the game.

Errors and Warnings

Closing an attribute that has not been opened results in a warning, but does not modify the final set of attributes for a line.

Failing to complete a tag (e.g. opening a tag with < but not providing the trailing >) will result in a warning, and the attribute not being generated.

Use Cases

Text formatting

Consider the following line:

Wow, <b>really</b>?

In this example, range of text from 5 to 11 would have the "b" attribute. The dialogue runner could then render the parts of the plain text within this range in boldface font; the specifics of how to do this depend on the engine. (In the specific case of using TextMeshPro in Unity, the strings  and  could be re-inserted into the plain text, causing TextMeshPro to format the text appropriately.

Character name extraction

Consider the following line:

Mae: Hello!

A dialogue runner can choose to display only the contents of the line after the character name by looking at the list of attributes, finding one with the name "character", and removing that substring from the plain text before displaying it.

In this example, the range of text from 0 to 5 would have the "character" attribute; removing this substring would result in the following plain text, ready to be shown to the player:

Hello!

Inline events

Consider the following line:

Hey! <mood=happy/> How are you doing?

A dialogue runner that displays each character in the line one at a time may detect the presence of a zero-length attribute "mood" at index 5, and change a character's facial expression after 5 characters have been presented.

Describe alternatives you've considered

The first alternative is to not add this feature. This would mean that developers continue implementing their own custom markup solutions.

Another alternative to the feature as specified is to add this feature in the compiler. This would make it a mandatory feature, and would require the implementation to be the same across all platforms, as well as creating new requirements for localisation.

I like this a lot, especially the name attribute being implicit. Implementing this on the DialogueRunner for localization seems to be the right way.

It would be best to implement this once the MVVM redesigned VO branch hits the develop branch, as the UI view will need to communicate to the Runner when it reaches a specific event.

This would pretty much solve almost every missing feature request I have!

Additionally, we could give users a way of associating their character GOs with the names they use in their yarn files.

For example, the Runner would know where Mae is in the scene making it easier for users to create a custom UI view that points a speech bubble to Mae. Additionally, the Runner would know Mae's AudioSource so the voice over view could use Mae's AudioSource for playback (allowing the sound designer to make character specific settings or have positional audio).

I agree that offering a basic standard for these features is overall positive and outweighs the negatives. I'm also pro HTML syntax.

If you want more elaborate character / speaker support in Unity, as @Schroedingers-Cat proposes, then you might want to look at how Fungus does things: there's a separate "Character" MonoBehaviour that you put on a game object, with some basic common settings like text color and audio. The Fungus equivalent of the dialogue runner automatically detects these character objects and binds them, though I imagine Yarn Spinner Unity would prefer manually assigning Characters to an array in the inspector? Either way, users could easily expand the Character class to expose more metadata. http://fungusdocs.snozbot.com/telling_a_story.html#characters

there's a separate "Character" MonoBehaviour that you put on a game object, with some basic common settings like text color and audio. The Fungus equivalent of the dialogue runner automatically detects these character objects and binds them, though I imagine Yarn Spinner Unity would prefer manually assigning Characters to an array in the inspector?

Thanks for the implementation example, I especially like the automatic binding between the character identifying MB and the DialogueRunner. I had a similar idea so seeing how fungus went about this feels like it'd be a good approach.

This would be brilliant!

I think this is a great idea—especially the implicit character tag— and I did have some concerns around semantic tags which I passed onto desplesda via other channels beforehand that have already made their way into this one, so awesome. I do have a few little concerns, none of which are massive deal breakers or anything, and some are a bit of "old man yells at clouds" so take them with a grain of salt.

Delimiter Choice and Line Processing

I understand the desire to use < and > as the delimiters given the obvious link to html but this creates a collision between the << control delimiters and the < and > operators. Doubly so in that inline expressions can also be used to build up values inside these tags.

This means statements like Whoa! <soundeffect=shake.wav/> <camerashake={$count > $trigger}/> An earthquake! while can be parsed only requires a tiny mistake (such as missing the closing }) and depending on implementation details might not error. Or mistyping a control statement as <if $someVar> now has to at least be attempted to be understood as a markup line instead of a mistyped if.

Not saying that these can't be worked around or detected just that the line handling process will need to be explicitly stated now, instead of implicitly like it is currently. So the process will have to explicitly become (with some simplifications as to how the parser works (ie steps 1 and 2 happen together):

Read a line in from the .yarn source
Determine expressions in the line
Splat expression results into the line
Modify line based on format functions
Handle any markup attributes inside the line

There is also some concerns here as to localisation and markup. We need to work really hard to let localisers know the rules for our markup so that if they need to change or remove them they can do so in a way that maintains intent.

This also conflicts with TMP auto handling of common html tags, the proposal even says "could be re-inserted into the plain text", this won't be necessary if we use different delimiters. That said unsure if we want to encourage mixing and matching markup styles so it might be better to always take over replacement for TMP regardless.

Ambiguity

"Unlike in HTML, attributes may be closed in any order.", I don't understand the purpose behind this and why it is desirable? Doesn't this just have the potential to lead to ambiguity?

Why, <wave=5>Hel<size=2>lo</wave></size>! is fairly easy to disambiguate but what about Why Hel<size=2>l<size=5>o!</size>?</size>? How attributes should the ! have, what about the ?? How are overlapping same attributes handled?

This is compounded with "unclosed attributes are automatically closed" rule which I am also confused as to its utility. So Why Hel<size=2>l<size=5>o!</size>? under this rule is valid and while I could make some assumptions (and codify them) about my earlier example above, I now have no idea which <size> has been closed here. Are both closed when the </size> tag is reached?

View Implementations

For this to be of use we will need to ensure that we have a few common tags already handled. Both to honour the we'll handle the parsing for you promise and so we can point to them as the basis for building your own custom ones. Which should we pick?

Raw Text

Related to the above, while I love the plain text and attributes part of this I think we probably also want a raw attribute. This would be the line post expression/formatters but pre-attribute extraction. This is for those who want to use the default yarn views but don't want to have tags or want to handle them themselves in specific uncommon ways. I suppose an alternative to this would be to create multiple default views, one that uses the tag system and one that doesn't.

Terminology overlap

Far more importantly, while not a technical concern the overlap in terminology between the tags header attribute, the #hashTag line tags, and soon the <markup> tags means we will have three completely different elements that are all called tags. I think we need to be diligent in our terms when talking to users and in the docs that these are markup, not tags. Or that the hashtags and tags attribute are something else.

Functionality Overlap

Similar to the above, not a technical concern but more a practice one. Should people be going:

<<shake 5>>
woah earthquake

or <shake=5>woah earthquake</shake>? Does it change if it is a visual effect vs an audio effect?

Opinions

I think we should choose a different delimiter or if people really think < works the best (I agree) perhaps we move away from << and >> seeing that 2.x can be a breaking change. This will require getting people to change their existing yarn files, either manually or perhaps this can be a feature built into the new console, a conversion between 1.x and 2.x.

I think we should require closing of unclosed tags, or at the very least throw a warning of unclosed tags on line blah.

I think we should codify and document the rule of last opened tag is first closed to resolve nested tag ambiguity.

I think we need to pick some terms (I am a big fan of markup) and stick to them, as well as create design practices that we should tell people to follow.

I just had another thought that occurs to me around the implicit character attribute. In the example:

Mae: Why, hello there!

This gets converted into:

Plain Text: Mae: Why, hello there!
Attributes
- Attribute:
  - Start: 0
  - Length: 5
  - Name: character
  - Properties:
  - name = Mae

If for example I want to go Mae <angry />: Why, hello there! what does this become? Is it a character attribute with a name of Mae <angry /> or does the implicit character get lost and we have a zero length position 4 attribute of angry?

The solution to this is easy, tell people to move the <angry /> after the : but we need to describe the markup parsing process so that we can understand these sorts of errors should they occur.

Related to this, the implicit character is equivalent to writing

<character name="Mae">Mae: </character> Why, hello there!

So if I write out

<character name="Mae", angry=true>Mae: </character> Why, hello there!

because I want to bundle all my character attributes up how does the implicit character work?

I still really like the idea of the implicit character attribute, but its a special case and special case = edge cases and we want them stamped out and fully explained for each one so there is no ambiguity either for those who have to implement such features or for the users who want to use the system.

I think we should choose a different delimiter or if people really think < works the best (I agree) perhaps we move away from << and >> seeing that 2.x can be a breaking change. This will require getting people to change their existing yarn files, either manually or perhaps this can be a feature built into the new console, a conversion between 1.x and 2.x.

I like that as I imagine it will increase readability. OTOH, could this motivate people to write <set $variable to false> midline and should that work?

Regarding terminology overlap, maybe we could rename header tags to Node ID and linetags to Line ID. That seems to fit better and allows us to name these <wave size=5> things tag which is what they look like.

@McJones, thank you for such a well thought out set of replies! I'll reply to each in sequence. I've paraphrased your replies with what I believe is the core of the issue - please let me know if I've misinterpreted you.

Delimiter Choice and Line Processing

Using < and > creates a collision with << and >>.

As far as the parser goes, there's no ambiguity between < and <<, since we use predicates to do look-ahead. Disambiguating < and << is already necessary in free text, so there's no additional work or complexity needed for the parser. For inline expressions, we enter a separate lexing mode already in which markup doesn't exist, so the problem doesn't appear there either.

The more syntax we add, the more opportunities for syntax errors or mistakes.

Users who are already adding their own markup will already potentially encounter this kind of error; by defining our own, we can catch the errors as part of the larger parsing system. For example, syntax errors in which a delimiter is missed is already a class of error - I agree that we need to do a much better job with our error messages, but that's an issue that's independent of this.

For mistakes that don't lead to a formal syntax error, we can add warnings. For example, in the specific case you mentioned where the user might accidentally type <if $foo> instead of <<if $foo>>, I recommend detecting the use of reserved keywords as attribute names and emitting a warning (or an error).

We'll need to formalise the parsing process into a two-part parse, where the first part parses to bytecode, and the second part does run-time line processing.

Agreed. That's what's happening with inline expressions and format functions, as of v1.1.

We'll need to create guidance for localisers, so that they know how to work with it.

Agreed. We currently have a page on the localisation features, but it's written for developers, not translators. We should create a dedicated page for translators and localisers.

If we used different delimiters than < and >, we wouldn't need to remove and re-add TextMeshPro tags.

True, but TextMeshPro isn't the only text solution out there, and as you noted, mixing-and-matching delimiters creates a headache for users.

My thoughts on which delimiters are the most appropriate ones to use are guided by the following considerations:

HTML-like tags are reasonably well-known as a formatting syntax.
Single-character delimiters are easier to type and parse than double-character delimiters.
We're currently using {} and [] for existing formatting functions.

Ambiguity

Why do we want to let attributes be closed in any order?

Why do we want to let attributes be implicitly closed?

This syntax isn't intended to be one for creating a document structure, but rather for associating attributes with runs of text. Therefore, a string like 123 has no ambiguity (1 is bold, 2 is bold and italic, 3 is italic). (As an aside: when given this string, WebKit silently rewrites it into well-formed HTML with the same formatting.)

These aren't nested elements, like HTML is - instead, these are start and end markers.

I do see an argument of "you're claiming HTML-like syntax but breaking expectations set by HTML", but HTML at this point is not intended to be written by hand, whereas this syntax is, and I feel justified in making syntactic concessions to make it easier to write by hand.

Therefore, requiring tags to be closed in a FIFO order, and requiring tags to be closed before the end of a line, adds work for writers to resolve an ambiguity that doesn't actually exist in the parser.

What happens when we have nested attributes of the same type, but with different properties?

Good question, and a gap in my initial proposal.

There are two ways we could address this:

Additional attributes stack up on existing ones. A range of text may have multiple attributes of the same type.
Additional attributes interrupt and replace existing ones. If a range of text has an attribute, and another of the same type is encountered, the first attribute ends and is replaced by the second; if there's more text after the second attribute ends, the first attribute is restored.
Make it an error to have multiple attributes with the same type.

In both the first and second options here, I propose that closing an attribute where there's potential ambiguity on which one to close, always closes the most recently entered attribute. That is, in <a=1><a=2></a>, the </a> is closing the <a=2>. This addresses your query of what should happen when only a single attribute is closed before the end of a line.

For the example you gave:

Hel<size=2>l<size=5>o!</size>?</size>

For option 1, this would be interpreted as:

Plain text: Hello!?
Attributes
- Attribute:
  - Start: 3
  - Length: 4
  - Name: size
  - Properties:
    - size = 2
- Attribute:
  - Start: 4
  - Length: 2
  - Name: size
  - Properties:
  - size = 5

For option 2:

Plain text: Hello!?
Attributes
- Attribute:
  - Start: 3
  - Length: 1
  - Name: size
  - Properties:
    - size = 2
- Attribute:
  - Start: 4
  - Length: 2
  - Name: size
  - Properties:
  - size = 5
- Attribute:
  - Start: 7
  - Length: 1
  - Name: size
  - Properties:
    - size = 2

The first option is simpler to parse and easier to explain, and is a little less opinionated about what multiple 'nested' attributes mean, but requires the game to make the decision on what to do about it.

The second option is harder to parse, and we'd need to explain the logic to end users, but requires less work on the part of the game - any ambiguity is dealt with at the parse level.

The third option is easiest for us to do, but limits what the user can do.

I don't really know which of these three options is the best. Thoughts?

View Implementations

We should have some common tags built-in, so that users don't have to do all of the work for common tasks. Which should we pick?

Agreed. The tags that work will depend on how the lines are delivered, and not every user will be using a delivery method that has a conceptual mapping for things like "bold text". That said, I'd like Yarn Spinner to be a bit of a 'batteries included' package for common tasks, so I think adding this is a good idea.

What I suggest is, in the dialogue views that we provide in our distribution in Yarn Spinner for Unity, we'll implement things like the following:

TextMeshPro-inspired formatting, implemented by inserting the equivalent TextMeshPro markup as per their docs.
Character animation: shaking, waving, hopping, scaling, (arbitrary animations via a curve?)
<event="EventName"> - Dispatches a Unity event from the view, passing a string parameter.

Raw Text

We should have a way to represent a range of text that's not parsed for markup.

Agreed. <raw> would work; TextMeshPro uses <nomarkup>; XML uses <!CDATA[[. I think raw is the best case.

This would need to be an edge case, since it would put the markup parser into a state where it doesn't perform any markup processing until it reaches the string </raw>, </>, or the end of a line's content.

Terminology overlap

We need to be careful about overloading the use of the term 'tag'.

Agreed. How about standardising on this terminology:

The tags header are tags, or node tags. (This header is present in user data already, and changing it would be difficult, so we'll leave it as-is.)
#line: items and similar #-prefixed strings are hashtags.
<markup> and </markup> syntax elements are markers that indicate the start and end of an attribute.
<markup/> is an example of a stand-alone marker.
The presence of markers in a text is markup.

Functionality overlap

Under what circumstances should we recommend that people use commands instead of attributes?

Commands are stand-alone content that's independent of the delivery of a line. Attributes and markers are tied to the line itself.

For the screen-shake example you gave:

A command is more appropriate when the screen starts shaking before the line begins
An attribute is appropriate when some or all of the line should animate with a shake
A stand-alone marker is appropriate when the screen should begin shaking during the delivery of the line.

Opinions

We should use a different delimiter to <, or replace the command delimiters with something different to <<.

I don't think I agree here. The < and << are unambiguous when parsed, and are visually distinct. I feel that users are used to the << syntax in Yarn, and that < is the most appropriate delimiter for markup, following HTML's example.

That said, do you have a proposal for what changing the << and >> delimiters in Yarn should look like?

I think we should require closing of unclosed tags, or at the very least throw a warning of unclosed tags.

I disagree here too! There's already an unambiguous end to a line at the line-end character, and we'd be asking users to write additional syntax that I don't think the language requires.

I think we should codify and document the rule of last opened tag is first closed to resolve nested tag ambiguity.

Disagree here also, but I outlined my reasoning above.

I think we need to pick some terms, as well as create design practices that we should tell people to follow.

Agreed, as per reasoning above!

If for example I want to go Mae <angry />: Why, hello there!, what does this become?

The stand-alone marker <angry /> would be parsed out. The remaining plain text would be Mae : Why, hello there! - note the space after Mae is preserved as written. With all of the markup removed, the implicit attribute character would then be generated.

The resulting parse is:

Plain Text: Mae : Why, hello there!
Attributes
- Attribute:
  - Start: 0
  - Length: 6
  - Name: character
  - Properties:
  - name = Mae
- Attribute:
  - Start: 4
  - Length: 0
  - Name: angry

If a game wanted to use markers inside the implicit character attribute, that's absolutely allowed, but the game would need to do the additional work of recognising and it.

Something that we could do is to not generate the implicit character attribute if one is present in the line. What do you think?

(From @Schroedingers-Cat) If we changed the command delimiters to <, and users tried to put a set command in the middle of the line, would that work?

Variable logic can't be done in the middle of a line, because Yarn Spinner delivers the entire line to the game and then awaits the signal to continue execution. This means that the variable wouldn't be able to change state halfway through, because line delivery is an atomic operation to the virtual machine.

What could be done is something similar to how inline expressions work, where the result of the expression is computed before the line is delivered, but that is potentially confusing when the set is halfway through a line.

For these reasons, I don't think that putting control logic like this in the middle of a line is a good idea, and I don't think that we should try to unify the delimiters.

Updated my previous comment to include a more complete description of how and when the implicit character attribute is parsed when markup exists inside the 'character name' range.

This syntax isn't intended to be one for creating a document structure, but rather for associating attributes with runs of text

Ah ok, with the html-ness being thrown around I assumed the aim was for a tree structure of attributes but if that's not the case then a lot of what I said is immediately gone as a concern. We do still need to decide how overlapping attributes work but this is much simpler than before and realistically any approach works well.

There's already an unambiguous end to a line at the line-end character, and we'd be asking users to write additional syntax that I don't think the language requires.

Ok then, so that is why we are now also getting rid of the closing >> on commands and the ]] on jumps right? Actually thinking about it we can probably also get rid of the the closing } on inline expressions too. The not closing a tag is a thing html has (and kinda never did, just the earlier browsers allowed it before it was an actual spec) but you already said we aren't trying to be html here. This invites errors and user misunderstanding in the same way as C does with allowing brace-less ifs. Symbol elision in any programming language has to be extremely consistent and only for cases where there is never any way to interpret it as user mistake (see the massive debates about implicit return and comma elision in Swift for good examples of the yay and nays around this).

As to the < vs << vs etc debate this one is less thought out and more "it just feels off to me" in a lot of the use of the poor overloaded <. It isn't that we can't disambiguate, we obviously can, it is just making the parser perform more work and increase the possibility for user error because they can type the wrong number of chevrons. Yes this can be helped with better error messages but it can also be side-stepped almost totally by reducing the overall symbol collision space. Yes the compiler figures it out but it does so using predicates which are known to slow down parsing in ANTLR (and I imagine in every parsing tool) and also harms the portability of everything because the predicates will also need to be rewritten.

If I had complete control I would first get rid of the [[ jump statement and make the format functions use ( instead, then you free up [ and ] as the markup symbols (which matches well with the BBCode which while not HTML level popular is pretty darn popular and closer matches how the markup works anyways) and then make jumps either <<nodeName|Text To Show>> or <<jump nodeName Text To Show>>, (I slightly prefer the latter) but that's just me. Or maybe just get rid of the jump entirely and repurpose the -> option symbol. This way we keep the << as the only control statement symbol and we keep (, { and [ for in-line stuff.

These are some good points, particularly about automatic closing of attribute runs. I'm fine to drop this idea from the proposal.

I agree that using [ and ] would be a good alternative to < and >. I'm a bit more wary of using round parentheses, because I expect that that's a kind of enclosing delimiter that users are more likely to want to use in their text.

I wonder if we could harmonise this a bit by adjusting format functions to be a kind of markup? Something like, instead of:

[plural {$pies} one="pie" many="pies"]

we could do:

[plural value={$pies} one="pie" many="pies" /]

From a parser complexity standpoint, we'd be folding two features into one, and removing the need to disambiguate a < from a << (both of which can appear inside a line, for example as a conditional at the end of a shortcut option). The downside would be that we're changing the format of a fairly recently-added feature, but maybe we can provide tooling to work around this (upgrading existing syntax like how Swift does it).

I'm not too wary of changing syntax even for a new feature, 2.x is a chance for a hard break, might as well take advantage of it.

@McJones and I had an off-GitHub conversation, and agreed to go ahead with this feature as per where we've ended up in this thread.

For the question of how deal with multiple overlapping attributes of the same type, we've settled on allowing a range of text to allow duplicates, and to leave the rest to the game. (Possibly a future linter could optionally flag it as an issue.)

This feature has now been merged into develop. Compatibility with the new API has been added to the develop branch of YarnSpinner-Unity.

Closing this issue now; thanks to everyone who contributed to the conversation! The feature, and Yarn Spinner overall, is better for your input.

YarnSpinnerTool / YarnSpinner