Add substitution format specifier

[salyu9]

This is a feature I've currently used in my own project and I wonder if it is appropriate to open this PR. Also more details need to be discussed to make it complete.

This feature add a formatter (or format specifier) syntax to substitutions in lines like many programming languages' built-in libraries do:

Number: {86:0000}.   // will be "Number: 0086." with proper format function.

I added a substitutionFormatterMode to the lexer and let StringTableGeneratorVisitor copied text from that mode to the generated string. And leave that as it is to be handled by the format function. Besides that, a properly object[] RawSubstitutions is added to Line class to be used in format function.

Motivation

• It's a common usage to add format specifier to format strings in variety of programming language such as C#, python and even C++20. It would be convenient to have this feature in yarn scripts.

• Format specifiers allows users to make the subsitution more customized, such as dealing with plurals in another approach like using SmartFormat: "There {0:plural:is 1 item|are {} items}.". Also some hack can be made like evaluate something that cannot be handled using YarnSpinner expressions. I've using this hack to make my {$characterName:genitive} refreshing when language switched. Hard to say allow more hacks is good or bad though.

• Unity also uses format specifier approach to deal with plurals in their official localization package now (with a modified version of SmartFormat). Supporting format specifier can let games using YarnSpinner and official package keep a same syntax of plural handling, that makes translators easy.

Problems I'm not sure my approach to implement this feature is good practice and may need some suggestions.

• I'm not sure whether the C# alignment syntax $"{52,-10}" (10-character left-aligned) need to be supported. I think the answer is yes because SmartFormat which Unity uses is using that. So I made a somehow strange inplementation (allows expression COMMA (+|-)? NUMBER in line_formatted_text rule). Another approach that other programming language uses like {:<10} is just a normal format specifier so that is auto supported.

• I added object[] RawSubstitutions property to keep backward compatibility. But I haven't implemented the new Dialogue.ExpandSubstitutions() method. It seems the original ExpandSubstitutions() is error-tolerant that just replace in-range placeholders and leave those our-ranged placeholders untouched. Should I make a similar format function that keep placeholders and formatters untouched if the formatting process fails? In my project I just throw exceptions which will have different behaviour to the original ExpandSubstitutions().

• The original Dialogue.ExpandSubstitutions() uses string[] that cannot be formatted again. Should I make a breaking change or just add a new expand method?

• The numbers in yarn values are doubles not ints, so int format specifiers are not available. Maybe this should be noticed in documentations.

Other information There's an antlr.sh script that download antlr-4.7, but the previous C# files are generated by antlr-4.9 which may be the bundled antlr version of vscode plugin. I used the latter to generate C# files. Also there are some test-cases fails on Windows due to the newline difference which is used by StringBuilder, maybe the test code need some fix.

[salyu9]

I tried to make a backward compatible format function that keeps placeholders untouched if failed to substitude (index out of range or invalid format specifier), but it's a bit complex and I doubt the necessity of keeping the compatibility. It will be easy to just return the source text if string.Format(localizedText, rawSubstitutions) throws exception. It is a really small breaking change and users that want more customizations will likely to use their own format functions (like SmartFormat or Unity.Localization).

[McJones]

Hey-oh,

So this is cool, I guess my first concern however is we haven't actually defined how interpolated text is to be displayed so without essentially base line of how things are to be shown providing formatting control is gonna be undefined. That's not a deal breaker, more "oh I guess we should document how things should show by default".

That aside I do wonder though if this is not done better via markup instead of overloading the interpolation syntax. The markup already allows a decent chunk of this, especially around plurals case. I do however think there is a need for some markup that allows numeric formatting, or maybe even a generic formatter markup that passes a format string down into something like SmartFormat.

I guess in the end I wonder what does this give over a new replacement markup that passes the format string to another system instead? Not saying that this is a deal-breaker just curious as to the best way to make this work.

[salyu9]

Thanks for replying!

I'm sorry that it seems I made the motivation explaning too heavily on the plural examples. The markup works really perfectly for plurals and other situations, and I'm not trying to make the format specifier to replace it. Actually I don't think YarnSpinner should support the plural format specifier by default as another way to do localizations besides the markup approach, that will make newcoming users confused.

What I want to achieve is adding a format specifier syntax support, which may just have basic function to format numbers with implementation-defined behaviour (most likely to be implemented as string.Format in YarnSpinner-Unity). The format handler can then be made customizable for advanced users so that they can do something with the format specifier, such as embed YarnSpinner to their games with the same plural syntax, if they are already using format specifier approach. It's a small syntax supporting that can enable more customizations.

I think a good way to do it might be like this:

• Support format specifier in core to be capatible to possible usages. In this PR I made a syntax that can take any character except '}', escaped '{{' and '}}, or '{}' that can recurse format specifier, which is needed by SmartFormat. This is also a part I worried if I'm using antlr correctly, please review there if we decide to merge this PR.

• Document in the "Writing in Yarn - Variables": add an information box that mentions: You can also use format specifiers in the braces like <<set $number to 3.141592>>number is {$number:F2} // will be "number is 3.14", the behaviour of the format specifier is implementation-defined. If you are using official Yarn Spinner C# runtime, you can use the same specifiers for floating numbers like string.Format in C#, see String.Format for more information.

• Expose a delegate CustomSubstituionHandler from DialogueRunner which falls back to string.Format (or wrap it in Yarn.Dialogue class), which can be set by end users. I can add this via PR to YarnSpinner-Unity also.

One more thing to worry about is whether to just use string.Format. The current search&replace implementation will make translation like {0} and {1} be 123 and {1} if the subsitutions contains only one item 123. If replaced by "try string.Format and catch" it will be simple and easy to maintain, but will be a small breaking change as I mentioned above, the result will be {0} and {1} with 123 not substitude.

The following is not important. My usage is rare and a bit tricky that make using markup more difficult than normal. Maybe it's so rare that it's not worth mentioned but if you are curious about: I'm making a card game that often need to describe a card function like Draw {$count} card(s). The count need to be highlighted in a different color, so after all process it should be look like Draw <color=blue>1</color> card / Draw <color=blue>2</color> cards (upgraded version of this card). Things get worse because other things can buff this card to let it draw more cards, in this situation the count need to be highlighted using a different color like green. I made my custom formatter to do the colorization and plural handling at the same time when substitude, which made the format string as simple as above like Draw {$count:plural one=...} ($count is not a simple number but a value-baseValue pair that will be colorize by the format handler). If I use the built-in markup, the text after substitution will be Draw [plural value="<color=green>2</color>" one=...] which is invalid. There's a way to do it via custom markups like Draw [plural-colorized value="{$count}" base="{$baseCount}" one=...], but I'll need to make a lot change to existing translations and it's also a little confusing or complex to translators. I firstly tried to use the markup syntax for this situation but gave up and turned to use my previous implementation, and made the dialogue part of the game to use similar plural syntax.

[st-pasha]

It's a bit unclear whether number formatting needs to be part of the core language syntax. It doesn't seem like this would be a commonly-used feature, and there are alternative ways to implement the same:

• declare a user-defined function such as format("%.3f", $number);
• declare the same function but as a built-in library function.

Then you could write something like

Number: {format("%04d", 86)}

to produce a line Number: 0086.

[McJones]

Closing this in favour of #348, thanks for the PR.

By the way we don't often see people making changes directly to the grammar and you did an excellent job with those changes, so while we aren't accepting this PR it is an absolutely amazing one and look forward to future PRs from you.