Proposal: Smart Variables

[desplesda]

Introduction

This proposal adds support for 'smart variables', which are read-only variables whose value is determined by evaluating an expression at run-time, rather than by storing and retrieving a value on disk or in memory. Smart variables are declared in Yarn scripts, and are accessible from inside Yarn scripts and and from outside Yarn (such as from a Unity C# script).

Note This language feature is part of the 'line groups' feature. A work-in-progress implementation of this feature is currently available on the features/line-groups branch.

<<declare $PieShouldAppear = $HasMetBaker && $Door_Unlocked>>

<<if $PieShouldAppear>>
    Baker: Enjoy your pie!
<<endif>>

Rationale

There are several reasons for this feature.

Expression Simplification

Developers store game state in Yarn variables, like "door unlocked" and "has met the baker". A common requirement is to have story progression depend upon multiple variables at the same time: 'door unlocked and has met the baker'. This is expressible as a boolean expression ($Door_Unlocked && $HasMetBaker), but this becomes unwieldy when the expressions become more complex, as the Yarn scripts become more difficult to read, and the opportunities for errors increases.

External Access

Dialogue is not the only part of a game that can depend upon a more complex expression. Often, the results of a conversation affects the overall state of the world. For example, imagine that when the door is unlocked and the player has met the baker, a pie appears on the table. In order to determine whether the pie is visible or not, the game needs to be able to get an answer to the question of whether both the door is unlocked and the player has met the baker.

Proposed solution

The overall direction of this feature is that smart variables can be read by any part of the game, including outside of Yarn scripts, just like a regular ('stored') variable.

This feature makes the following changes to the Yarn Spinner compiler and runtime.

Compiler Changes

• declare statements now accept an expression, rather than requiring a constant value. Variables declared in this way are called 'smart variables'.
  • Expressions in a declare statement may be any legal Yarn expression, and may reference other smart variables.
  • It is a compiler error for two or more declare statements to create a dependency loop (for example, it is an error for A to depend on B, and for B to depend on A).
• Smart variables are read-only. It is a compiler error to modify a smart variable in a Yarn script, and it is a runtime error to modify a smart variable from outside a Yarn script (such as by calling SetValue on a variable storage object)
• For each smart variable, a Node is created that contains the bytecode for evaluating that expression.
• During compilation, the compiler keeps track of whether a variable is a smart variable or a stored variable.
  • When a smart variable is accessed, the expression is evaluated.
  • When a stored variable is accessed, its value is retrieved from variable storage or from initial values.

Runtime Changes

• Variable storage objects (for example, InMemoryVariableStorage) keeps a reference to a 'smart variable evaluator', which is an object that can determine the runtime value of a smart variable. (This is typically the Dialogue object that owns the variable storage.)
• When a variable's value is requested, the variable storage object first determines if the variable is a smart variable, or a stored variable.
  • If the variable is a stored variable, its value is fetched from storage or from initial values.
  • If the variable is a smart variable, the smart variable evaluator is queried for the value.

The smart variable evaluator produces the value of a smart variable by first finding the node that contains the Yarn bytecode for the smart variable, and then evaluating that bytecode. This produces the result, which the smart variable evaluator returns.

Detailed design

The declare_statement rule in the grammar is updated to permit expressions, whereas it previously permitted only constant values.

declare_statement
    : COMMAND_START COMMAND_DECLARE variable OPERATOR_ASSIGNMENT expression ('as' type=FUNC_ID)? COMMAND_END
    ;

For each variable declaration where the type is non-constant (i.e. a runtime-expression), the compiler will emit a Node containing bytecode that evaluates the expression, leaving the expression result at the top of the stack. (This value is then popped by a smart variable evaluator as the final step of determining a variable's expression.)

The IVariableStorage interface is modified to require two new properties:

// Accesses a Yarn Program in order to determine whether a variable is stored, smart, or unknown.
Program Program { get; set; }

// Accesses an object that can determine the runtime value of a smart variable.
ISmartVariableEvaluator SmartVariableEvaluator { get; set; }

ISmartVariableEvaluator contains one method:

// Attempts to evaluate a smart variable of type T. Returns true if the evaluation succeeded (and sets 'result' accordingly). Returns false if 'name' is not a valid smart variable, or if the type of the variable is not convertible to T.
bool TryGetSmartVariable<T>(string name, out T result);

Backwards Compatibility

This feature makes purely additive changes to the Yarn language grammar; previously, declare statements required a constant value, and this is now expanded to permit expressions.

Finally, the Yarn Spinner Language Server, Try Yarn Spinner, and the Yarn TypeScript runner will need to be updated with the updated grammar and functionality.

We will mitigate the impact of these API changes by updating existing types, such as InMemoryVariableStorage, to have this support built-in (and updating existing code accordingly). These types are frequently subclassed by end-users, so they will receive this functionality as a built-in feature and will not need to make any changes.

Alternatives considered

Other considered alternatives include:

• Implementing all smart variables in the game host (such as in C#), and not in Yarn
• Adding new syntax for declaring 'functions' in Yarn script

Both of these approaches introduce more work required for end-users, and in the case of adding new function syntax, significantly change the architecture of Yarn Spinner. Accordingly, a more lightweight approach is preferred.

Acknowledgments

We'd like to thank the Yarn Spinner Discord members for their thoughts on the design of this feature; in particular, Khan-ali Ibrahim (@KXI-System) for the term 'smart variable', Robert Yang (@radiatoryang), and @dogboydog.

[KXI-System]

As I mentioned on the Discord server, the ability to also leverage Yarn Functions with Smart Variables opens up more ways to incorporate data from the game into the Yarn story so I really do like this proposal. However I see a few issues in terms of usability that I want to highlight.

The first is an issue of syntax, smart variables look exactly like normal variables. This isn't too much of an issue when defining them, but just reading a normal story it can get confusing to distinguish the two. Of course this can be alleviated with proper naming conventions, so instead of $PieShouldAppear its $S_PieShouldAppear, but this could also be fixed on the language level. While this isn't the biggest issue by itself, it might be worth considering with my other issues.

Which would be the behaviour when switching Yarn Projects, with normal variables a general declare statement for its type would be enough but Smart Variables need more information. This could mean that the function of a smart variable could change between projects, which would be an added feature, but also cause some possible unknowns like:

1. What happens when a Smart Variable defined in Project A has a different type than one with the same name in Project B?
2. What happens when a Smart Variable is accessed through the implementation but isn't declared in the project?
3. What if the Smart Variable was declared in Project A, the game then switches to Project C where there isn't a declaration because it isn't used, but scenario 2 happens?
4. What if a variable of the same name was declared as a smart variable in Project A, but a normal variable in Project D? What happens when a game switches between them?

For all these scenarios I expect errors, but 1 and 4 are human errors that will happen. My proposed solution for scenario 4 is to have smart variables have an extra $ character, so $$PieShouldAppear will be a smart variable while $PieShouldAppear is a normal variable.

Scenario's 2 and 3 requires an understanding of how Smart Variables are evaluated, but it isn't an far fetch mistake to make if you assume smart and normal variables are identical, or that the evaluated value gets saved. For scenario 3 I would want some sort of feedback where the warning/error would mention if the smart variable had a definition in a previous project, but the current loaded one doesn't.

[dogboydog]

Can't regular variables also be different types between project A and project B? Maybe I'm thinking of it wrong, but I thought YarnProjects were basically their each their own domain, where variables and node names etc. could differ. I've only ever had one yarn project in my game, but I didn't think there was any special logic to carry anything over when switching projects so I'm wondering how this applies to smart variables in particular rather than all variables

[KXI-System]

That is true, I forgot scenario 1 could happen to normal variables too. Though if I'm understanding the logic of how Smart Variables would work, they should work without any problems contained in their own projects, just confusingly changing type in context to other projects. In comparison to normal variables where mixing types would be a more apparent problem.

[McJones]

I can definitely answer some of these with what I think should happen

1. What happens when a Smart Variable defined in Project A has a different type than one with the same name in Project B?

Same as what happens with any duplicate declaration, it's an error, nothing changes here in regards to smart variables.

2. What happens when a Smart Variable is accessed through the implementation but isn't declared in the project?

Smart Variables are intended to be a way to wrap up a lot of variables into one easier to interface with one. As such they must be declared in the project to be accessed externally. Another way to put it is "what happens if you access a regular variable that doesn't exist?" because the same thing should happen with Smart Variables, the variable store will go "I don't know what that is".

3. What if the Smart Variable was declared in Project A, the game then switches to Project C where there isn't a declaration because it isn't used, but scenario 2 happens?

Same as Scenario 2, the variable storage won't know what it is and will tell you as such.

4. What if a variable of the same name was declared as a smart variable in Project A, but a normal variable in Project D? What happens when a game switches between them?

This IMO is the same as Scenario 1, that it is a smart variable doesn't change that it is has been declared multiple times, it's an error to do this. Now the difference here is you did it at runtime but this is still an issue that persists without it being smart variables. Because you control the variable storage, swapping out the values of variables at runtime is in your hands. In the case of the WIP branch it will I think use the smart variable value because we check for variables being smart first and then fallback to store, I would have to double check to be sure.

Overall the way to think of smart variables in my mind is the same as computed properties in C#. Unless you dive into the inner workings of an object you won't know or even worry about them. In the case of Yarn Spinner because you have the ability to also control the memory you do have to at least consider them, but I am willing to bet most people use the built in variable storage in which case again they don't have to worry.

[desplesda]

This feature has been announced as part of Yarn Spinner 3.0. Accordingly, I'll close the issue here - thanks for the discussion, everyone!