tokay-lang / tokay

Tokay is a programming language designed for ad-hoc parsing, inspired by awk.
https://tokay.dev
MIT License
236 stars 7 forks source link

bug: Recursive generic definition fails to resolve #127

Closed phorward closed 2 weeks ago

phorward commented 7 months ago

This program

Assignment : @<Assignment, Expression, ext: void> {
    Ident _ '=' _ Expect<Assignment>  ast("assign" + ext)
    Expression
}

HoldAssignment : Assignment<HoldAssignment, Int>

HoldAssignment

generates the panic

thread 'main' panicked at src/compiler/iml/imlvalue.rs:102:27:
not yet implemented: Recursive resolve()
phorward commented 7 months ago

This is a problem with the current internal ImlValue design. fff4aa8575fcc5ef756be7d632b6266b68a08f83 already marked this problems some weeks ago. Possible solution is to remove ImlValue::Unresolved() and generally introduce a ImlRefValue, similar to ImlValue, which allows self-referencing and dynamic borrowing.

phorward commented 7 months ago

a10b45a paves the way for release v0.6.5, which supports generic parselets. This issue won't be resolved for this version, and must be resolved by a future refactor or redesign. This issue is currently referenced in the panic, in case somebody gets into it.

Current resolve for the problem presented above is to define HoldAssignment as a separate parselet:

Assignment : @<Assignment, Expression, ext: void> {
    Ident _ '=' _ Expect<Assignment>  ast("assign" + ext)
    Expression
}

# workaround
HoldAssignment : @{
    Assignment<HoldAssignment, Int>
}

HoldAssignment
phorward commented 5 months ago

Better example, currently tested with compiler-rework-test branch contents:

Assignment : @<Expression, Assignment: Assignment, ext: void> {
    Ident _ '=' _ Expect<Assignment>  ast("assign" + ext)
    Expression  ast("value")
}

HoldAssignment : Assignment<Int, HoldAssignment>  # endless recursion!
# HoldAssignment : Assignment<Int, HoldAssignment> Empty  # works!

# ast_print(Assignment<Int>)
ast_print(HoldAssignment)

A correct run of this program would look like this:

a=b=c=2
assign [start 1:1, end 1:8]
 assign [start 1:3, end 1:8]
  assign [start 1:5, end 1:8]
   value [start 1:7, end 1:8] => 2
a=2=3
assign [start 2:1, end 2:4]
 value [start 2:3, end 2:4] => 2
value [start 2:5, end 2:6] => 3

The problem is generally that an value_generic AST-node becomes an ImlValue::Instance, which is then directly turned into a ImlParselet. In the above example, HoldAssignment looks like a parselet, but it is just an alias for Assignment<Int, Assigment<Int, Assignment...>> with endless recursion, which cannot be resolved. Therefore, HoldAssignment must become its own parselet!

I did some tests in traverse_node_static to fix this, but it isn't satisfying enough, together with some side-problems (consumable detection, multiple error messages referring the same problem).

fn traverse_node_static(scope: &Scope, assign: Option<String>, node: &Dict) -> ImlValue {
    let emit = node["emit"].borrow();
    let emit = emit.object::<Str>().unwrap().as_str();

    if emit.starts_with("value_") && (emit != "value_generic" || assign.is_none()) {
        traverse_node_value(scope, node, assign)
    } else {
        // Handle anything else as an implicit parselet in its own scope
        let implicit_parselet = ImlParselet::new(ImlParseletInstance::new(
            None,
            None,
            traverse_node_offset(node),
            assign,
            5,
            false,
        ));

        implicit_parselet.borrow().model.borrow_mut().body = {
            match traverse_node_rvalue(
                &scope.shadow(ScopeLevel::Parselet(implicit_parselet.clone())),
                node,
                Rvalue::CallOrLoad,
            ) {
                ImlOp::Nop => return value!(void).into(),
                // Defined value call without parameters, or load becomes just the value
                ImlOp::Load { target: value, .. }
                | ImlOp::Call {
                    target: value,
                    args: None,
                    ..
                } if emit != "value_generic" => return value,

                // Any other code becomes its own parselet without any signature.
                body => body,
            }
        };

        ImlValue::from(implicit_parselet)
    }
}
phorward commented 4 months ago

This issue is still existing and should be kept open!