create interpreter vm

[eignnx]

it would be cool to create an interpreted low-level language for the Generator object.

this would allow for optimization passes such as inlining and constant folding.

the idea is:

• instead of constructing lots of Vec<OutToken>s, we will have one mutable String that is being built up incrementally. this should save on allocation.
• instead of a recursive algorithm, the generator will use effectively goto statements and a virtual stack.

Instructions

enum Instr {
    /// Pushes some raw text onto the sentence-string currently being constructed.
    PushStr(&str),

    /// For when the text to be pushed might contain `+`s. Should be optimizable to `PushStr`s.
    PushOutSym(OutSym),

    /// Randomly selects one element from its argument.
    Choose(&[Sentence]),

    /// Assumes the `DataVariant` rule-ref arguments are accessible somewhere ("in a register")
    /// and performs pattern matching on the `Case`s. The selected `Case` will subsequently
    /// be executed.
    CaseSelect(&[Case]),

    /// Returns the (now completed) sentence-string to the calling program, and exits the vm.
    Return,

    /// Looks up the value of a `DataVariable`, if no value is set, one will be chosen at random.
    GetVariableValue(DataVariable),

    /// Uses a "local value" to store a `DataVariant` in the current state. This is used when a
    /// `DataVariable` appears in a case pattern.
    SetVariableValue(DataVariable),

    /// Evaluates the `RuleDecl` and passes in the "local arguments" vector.
    RuleEval(&RuleDecl)

    // etc.
}

Optimization Example

Inlining (Example 1)

rule start = foo
rule foo = "FOO" bar
rule bar = -- ...

→

rule start = "FOO" bar
rule bar = -- ...

Inlining (Example 2)

rule start = "a" foo "b"
rule foo = "X" | "Y" | "Z"

→

rule start =
  | "a" "X" "b"
  | "a" "Y" "b"
  | "a" "Z" "b"

Constant Folding (Example 1)

this example assumes the ! capitalization symbol is implemented via #11 .

rule start = !"a" "baby" "cried" + "."

→

rule start = "A baby cried."