Meta design: Use JSON equivalent representation with source range as secrets

Alternative to https://github.com/PistonDevelopers/dyon/issues/182

The meta converter language does not support repeated conversions, which is required if you need some complex rewrite logic (e.g. proof assistants). This idea is an attempt to separate the parsing step and use a more general system for tree transformation.

Piston-Meta already uses a JSON converter to debug meta data. Since Dyon is very similar to Javascript, perhaps converting meta data to a Dyon data structure directly would make it easier to work with it? This structure is easy to debug, and because Dyon is dynamically typed it is easy to transform the structure gradually, without using too much memory.

Instead of using a meta converter language, one could generate the JSON equivalent representation of the meta data. The source range could be stored as secrets, thereby keeping the trace back to the source even under complicated transformations.

Example

You want to parse a custom format looking like this:

assets/test_data.txt

0 1 2
3 4 5
6 7 8

So, you write a meta syntax file:

assets/test_meta.txt

1 pos = [.w? .$_:"x" .w! .$_:"y" .w! .$_:"z" .w?]
0 doc = .l(pos:"pos")

In Dyon, you call load__meta_file:

data := unwrap(load(meta: "assets/test_meta.txt", file: "assets/test_data.txt"))
println(data)

This generates the Dyon data:

{
  pos: {x: 0, y: 1, z: 2},
  pos: {x: 3, y: 4, z: 5},
  pos: {x: 6, y: 7, z: 8}
}

To get the source range in characters:

println(sec(data[0].x)) // prints `some([(0, 1)])`

Unsolved problem: How to repeat meta data

In the example above, I assumed the following problem was solved:

The JSON representation in Piston-Meta is not correct for all input, since it replaces the same keys with new values when a sub rule is repeated within same parent node.

To solve this problem, it might require a change to Piston-Meta. It need to pass on information about data generated from repetition rules. Repetition rules are .l (lines), .r (repeat), .s (separated by).

One challenge is that recursion can also be repetitive when calling a sub rule without generating a new node. Perhaps this would give ideas to solve problems with left recursion?

Add support for secrets to `str`

Currently, Dyon only supports secrets for bool and f64. This is used in the mathematical loops.

To trace back text read from an input file, it requires support for str secrets in Dyon.

Add `sec` intrinsic to reveal secret of `bool`, `f64` and `str`

Get the secret of bool, f64 and str. Returns none() if there is no secret.

fn sec(any) -> opt[[any]] { ... }

This is required because the existing functions are type specific and does not work in all cases for parsing. why checks for true and where checks for non-nan. These are designed to be used with mathematical loops and detect logical bugs at run-time. Need the more general sec to do this on all parsed data.

Rewriting and transformation

The following strategies can be used to transform the Dyon structure into a target structure:

Visitor pattern with closures to walk over the tree
Rewrite nodes using depth-first
Converter language could generate Dyon => Dyon
Add if <pat> ?= <rexpr> { ... } for pattern matching

An idea is to add some sugar for pattern matching to the Dyon language, in order to make tree transformations easier to deal with:

x := [1, 2]
if [a, b] ?= x {
    println(a)
    println(b)
}

For functions that returns res, one could use <pat> ?= <rexpr>:

fn foo() -> res {
    x := [1, 2]
    [a, b] ?= x
    ...
}

PistonDevelopers / dyon