search

tokay-lang / tokay

Tokay is a programming language designed for ad-hoc parsing, inspired by awk.
https://tokay.dev
MIT License
236 stars 7 forks source link

Replace Tokay's internal parser by a parser generated from `tokay.tok` #93

Closed phorward closed 1 year ago

phorward commented 1 year ago

This is quite like a lizard eating its own tail, but yeah, it is!

This PR is a draft for finally fixing #66 and eliminates the maintenance of two parsers (parser.rs and tokay.tok).

It generates most contents of the file parser.rs by executing and parsing tokay.tok with a slightly modified version of its own. parser.rs contains a Rust-encoded version of the AST generated from a parse of tokay.tok, which can directly be compiled.

phorward commented 1 year ago

Error reporting is a mess with this, especially when using the expect-construct.

>>> esel(
Line 1, column 6: Expecting Object(Touch(")"))
>>> ekay +
Line 1, column 7: Expecting ImlParselet { offset: None, consuming: true, severity: 5, name: Some("MulDiv"), constants: [], signature: [], locals: 0, begin: Nop, end: Nop, body: Alt { alts: [Seq { seq: [Shared(RefCell { value: Call { offset: None, target: Parselet(MulDiv), args: None } }), Call { offset: None, target: Object(Touch("*")), args: None }, Call { offset: None, target: Parselet(_), args: None }, Expect { body: Call { offset: None, target: Parselet(Unary), args: None }, msg: Some("Expecting ImlParselet { offset: None, consuming: true, severity: 5, name: Some(\"Unary\"), constants: [], signature: [], locals: 0, begin: Nop, end: Nop, body: Alt { alts: [Seq { seq: [Call { offset: None, target: Object(Touch(\"-\")), args: None }, Not { body: Call { offset: None, target: Object(Touch(\"-\")), args: None } }, Call { offset: None, target: Parselet(_), args: None }, Shared(RefCell { value: Call { offset: None, target: Parselet(Unary), args: None } }), Load { offset: None, target: Object(\"op_unary_neg\") }, Call { offset: None, target: Object(ast), args: Some((1, false)) }], collection: true }, Seq { seq: [Call { offset: None, target: Object(Touch(\"!\")), args: None }, Call { offset: None, target: Parselet(_), args: None }, Shared(RefCell { value: Call { offset: None, target: Parselet(Unary), args: None } }), Load { offset: None, target: Object(\"op_unary_not\") }, Call { offset: None, target: Object(ast), args: Some((1, false)) }], collection: true }, Seq { seq: [Call { offset: None, target: Parselet(Rvalue), args: None }, Call { offset: None, target: Parselet(_), args: None }], collection: true }] } }") }, Load { offset: None, target: Object("op_binary_mul") }, Call { offset: None, target: Object(ast), args: Some((1, false)) }], collection: true }, Seq { seq: [Shared(RefCell { value: Call { offset: None, target: Parselet(MulDiv), args: None } }), Call { offset: None, target: Object(Touch("//")), args: None }, Call { offset: None, target: Parselet(_), args: None }, Expect { body: Call { offset: None, target: Parselet(Unary), args: None }, msg: Some("Expecting ImlParselet { offset: None, consuming: true, severity: 5, name: Some(\"Unary\"), constants: [], signature: [], locals: 0, begin: Nop, end: Nop, body: Alt { alts: [Seq { seq: [Call { offset: None, target: Object(Touch(\"-\")), args: None }, Not { body: Call { offset: None, target: Object(Touch(\"-\")), args: None } }, Call { offset: None, target: Parselet(_), args: None }, Shared(RefCell { value: Call { offset: None, target: Parselet(Unary), args: None } }), Load { offset: None, target: Object(\"op_unary_neg\") }, Call { offset: None, target: Object(ast), args: Some((1, false)) }], collection: true }, Seq { seq: [Call { offset: None, target: Object(Touch(\"!\")), args: None }, Call { offset: None, target: Parselet(_), args: None }, Shared(RefCell { value: Call { offset: None, target: Parselet(Unary), args: None } }), Load { offset: None, target: Object(\"op_unary_not\") }, Call { offset: None, target: Object(ast), args: Some((1, false)) }], collection: true }, Seq { seq: [Call { offset: None, target: Parselet(Rvalue), args: None }, Call { offset: None, target: Parselet(_), args: None }], collection: true }] } }") }, Load { offset: None, target: Object("op_binary_divi") }, Call { offset: None, target: Object(ast), args: Some((1, false)) }], collection: true }, Seq { seq: [Shared(RefCell { value: Call { offset: None, target: Parselet(MulDiv), args: None } }), Call { offset: None, target: Object(Touch("/")), args: None }, Call { offset: None, target: Parselet(_), args: None }, Expect { body: Call { offset: None, target: Parselet(Unary), args: None }, msg: Some("Expecting ImlParselet { offset: None, consuming: true, severity: 5, name: Some(\"Unary\"), constants: [], signature: [], locals: 0, begin: Nop, end: Nop, body: Alt { alts: [Seq { seq: [Call { offset: None, target: Object(Touch(\"-\")), args: None }, Not { body: Call { offset: None, target: Object(Touch(\"-\")), args: None } }, Call { offset: None, target: Parselet(_), args: None }, Shared(RefCell { value: Call { offset: None, target: Parselet(Unary), args: None } }), Load { offset: None, target: Object(\"op_unary_neg\") }, Call { offset: None, target: Object(ast), args: Some((1, false)) }], collection: true }, Seq { seq: [Call { offset: None, target: Object(Touch(\"!\")), args: None }, Call { offset: None, target: Parselet(_), args: None }, Shared(RefCell { value: Call { offset: None, target: Parselet(Unary), args: None } }), Load { offset: None, target: Object(\"op_unary_not\") }, Call { offset: None, target: Object(ast), args: Some((1, false)) }], collection: true }, Seq { seq: [Call { offset: None, target: Parselet(Rvalue), args: None }, Call { offset: None, target: Parselet(_), args: None }], collection: true }] } }") }, Load { offset: None, target: Object("op_binary_div") }, Call { offset: None, target: Object(ast), args: Some((1, false)) }], collection: true }, Seq { seq: [Shared(RefCell { value: Call { offset: None, target: Parselet(MulDiv), args: None } }), Call { offset: None, target: Object(Touch("%")), args: None }, Call { offset: None, target: Parselet(_), args: None }, Expect { body: Call { offset: None, target: Parselet(Unary), args: None }, msg: Some("Expecting ImlParselet { offset: None, consuming: true, severity: 5, name: Some(\"Unary\"), constants: [], signature: [], locals: 0, begin: Nop, end: Nop, body: Alt { alts: [Seq { seq: [Call { offset: None, target: Object(Touch(\"-\")), args: None }, Not { body: Call { offset: None, target: Object(Touch(\"-\")), args: None } }, Call { offset: None, target: Parselet(_), args: None }, Shared(RefCell { value: Call { offset: None, target: Parselet(Unary), args: None } }), Load { offset: None, target: Object(\"op_unary_neg\") }, Call { offset: None, target: Object(ast), args: Some((1, false)) }], collection: true }, Seq { seq: [Call { offset: None, target: Object(Touch(\"!\")), args: None }, Call { offset: None, target: Parselet(_), args: None }, Shared(RefCell { value: Call { offset: None, target: Parselet(Unary), args: None } }), Load { offset: None, target: Object(\"op_unary_not\") }, Call { offset: None, target: Object(ast), args: Some((1, false)) }], collection: true }, Seq { seq: [Call { offset: None, target: Parselet(Rvalue), args: None }, Call { offset: None, target: Parselet(_), args: None }], collection: true }] } }") }, Load { offset: None, target: Object("op_binary_mod") }, Call { offset: None, target: Object(ast), args: Some((1, false)) }], collection: true }, Call { offset: None, target: Parselet(Unary), args: None }] } }

Generally, repr() should be used here.