Runefile syntax for multiple inputs and outputs

[Michael-F-Bryan]

I'm trying to figure out the syntax for a Runefile where each stage might have multiple inputs and multiple outputs, and I was hoping to get some input.

Example

As an example, let's do some DSP preprocessing which takes data from the audio capability and passes it through a fftpreprocessor. Then on the other side we have samples from the accelerometer being passed through a normalize step to make the data independent of how far/hard you move the device.

Next we take the outputs from fft and accelerometer and send them to a model. This model sends a list of probabilities to the label proc block which then turns it to a UTF-8 string that gets sent to the main output.

We've also got a debug output which taps into fft , model, and label and you may want to disable independently.

That's a lot of words, but we're essentially trying to construct this pipeline:

Concept 1 - Modified Runefile

One syntax idea is to use --input some_stage and --input-type f32[42] for declaring a stage's input(s), with the corresponding --output some_stage and --output-type i8[1960] for specifying the output stage and its type. Under this model you can have multiple inputs but a single output (but that output may be copied to multiple stages).

This means we'd stop using angle brackets for input/output types because I think it'd be hard to adapt the PROC_BLOCK<input_type, output_type> syntax to multiple inputs/outputs without it being ugly or turning into punctuation soup (e.g. PROC_BLOCK<(i32[1920, 1080], f32[128, 3]), f32[4]> model ...).

FROM runicos/base

CAPABILITY audio SOUND --hz 16000 --output-type I16[16000]
CAPABILITY accelerometer ACCEL -n 128 --output-type F32[128, 3]

PROC_BLOCK fft hotg-ai/rune#proc_blocks/fft \
    --input audio --input-type I16[16000] \
    --output-type I8[1960]
PROC_BLOCK normalize hotg-ai/rune#proc_blocks/normalize \
    --input accelerometer --input-type f32[128, 3] \
    --output-type f32[128, 3]
MODEL model ./model.tflite --input audio \
    --input fft \
    --input normalize --input-type f32[128, 3] \
    --output-type f32[4]
PROC_BLOCK label hotg-ai/rune#proc_blocks/ohv_label \
    --labels=Wing,Ring,Slope,Unknown \
    --input model \
    --output-type UTF8

OUT output SERIAL --input fft --input model --input label
OUT debug SERIAL --input label

That would be a relatively minor change to the language syntax and probably take a couple hours of fiddling in the parser. Unfortunately it makes a Runefile context sensitive (a --input-type argument can only follow an --input and some arguments are only specific to certain commands) which isn't nice from a language design standpoint.

Concept 2 - Lua

Alternatively, instead of writing our own DSL we could leverage an existing one (e.g. Lua with our own custom functions):

audio = capability {
    kind = "SOUND",
    output_type = i16(16000),
    args = { hz = 16000 },
}

accelerometer = capability { kind = "ACCEL", output_type = f32(128, 3) }

normalize = proc_block {
    source = "hotg-ai/rune#proc_blocks/normalize",
    input = accelerometer,
    output_type = f32(128, 3),
}

model = model {
    filename = "./model.tflite",
    input = { audio, accelerometer },
    output_type = f32(4),
}

label = proc_block {
    source = "hotg-ai/rune#proc_blocks/ohv_label",
    input = model,
    output_type = utf8,
    args = {
        labels = { "Wing", "Ring", "Slope", "Unknown" },
    },
}

output = out { kind = "SERIAL", input = { label } }
debug = out { kind = "SERIAL", input = { fft, model, label } }

I'd maybe consider the above for a Runefile v3 seeing as it's quite a large change.

[Michael-F-Bryan]

Concept 3 - YAML

image: "runicos/base"

pipeline:
  audio:
    capability: SOUND
    outputs:
    - type: i16
      dimensions: [16000]
    args:
      hz: 16000

  accelerometer:
    capability: ACCEL
    outputs:
    - type: f32
      dimensions: [128, 3]

  normalize:
    proc-block: "hotg-ai/rune#proc_blocks/normalize"
    inputs:
    - accelerometer
    outputs:
    - type: f32
      dimensions: [128, 3]

  model:
    model: "./model.tflite"
    inputs:
    - audio
    - accelerometer
    outputs:
    - type: f32
      dimensions: [4]

  label:
    proc-block: "hotg-ai/rune#proc_blocks/ohv_label"
    inputs:
    - model
    outputs:
    - type: utf8
    args:
      labels: ["Wing", "Ring", "Slope", "Unknown"]

  output:
    out: SERIAL
    inputs:
    - label

  debug:
    out: SERIAL
    inputs:
    - fft
    - model
    - label

[Michael-F-Bryan]

@delimbetov liked the idea of using YAML for our Runefile format and proposed a couple tweaks to the document structure:

Hi, I think that using existing languages is gonna be easier for everyone. Personally I like the yaml one, but I would like it even more if blocks were split instead of all of them being part of the pipeline. E.g.

image: "runicos/base"

capabilities:
  audio:
    capability: SOUND
    outputs:
    - type: i16
      dimensions: [16000]
    args:
      hz: 16000

  accelerometer:
    capability: ACCEL
    outputs:
    - type: f32
      dimensions: [128, 3]

procblocks:
  normalize:
    proc-block: "hotg-ai/rune#proc_blocks/normalize"
    inputs:
    - accelerometer
    outputs:
    - type: f32
      dimensions: [128, 3]

pipeline:
  model:
    model: "./model.tflite"
    inputs:
    - audio
    - accelerometer
    outputs:
    - type: f32
      dimensions: [4]

  label:
    proc-block: "hotg-ai/rune#proc_blocks/ohv_label"
    inputs:
    - model
    outputs:
    - type: utf8
    args:
      labels: ["Wing", "Ring", "Slope", "Unknown"]

outputs:
  output:
    out: SERIAL
    inputs:
    - label

  debug:
    out: SERIAL
    inputs:
    - fft
    - model
    - label

[Michael-F-Bryan]

After talking with @meelislootus yesterday I think our plan of attack should be to make some small tweaks to the Runefile syntax (i.e. concept 1) in the short term because we've already started putting out marketing material and docs with our current Runefile syntax. Then a month or two afterwards we can start to phase it out in favour of YAML (concept 3).

The two formats will be feature-for-feature identical, so it should be trivial to read in an old-style Runefile and output the YAML equivalent.

Akshay and @kthakore, what are your thoughts? This is an easy thing to implement in code but considering the Runefile will be the primary way people interact with Rune on a day-to-day basis, changing its syntax/format could have larger ramifications from a business perspective.

[kthakore]

I think the Yaml approach fits well in the space given. yaml can also be syntax checked on the client side. I an in favor of it!

[akshr]

Pretty late to the game, sorry, but ya YAML for me as well :) - it draws the same orchestration crowd as well! I think the short term concept 1 makes sense too, but maybe we can be careful with the community? Because post launch if we need to change this we need to communicate really well.