tokay-lang / tokay

Tokay is a programming language designed for ad-hoc parsing, inspired by awk.
https://tokay.dev
MIT License
239 stars 7 forks source link

ASTs of `parser.rs` and `tokay.tok` are unequal #73

Closed phorward closed 2 years ago

phorward commented 2 years ago

There are still some differences between the grammars in the macro-based parser.rs and pure-Tokay tokay.tok. This issue relates to #66, and is a precondition to it.

In the commit 1263c1a1ea11b461405b172d5b8bb71f1fdd608f these two calls

$ cargo run -- -d examples/tokay.tok >/tmp/macro.tok.ast
$ cargo run -- examples/tokay.tok -- examples/tokay.tok  >/tmp/tokay.tok.ast

generate the following diff (diff macro.tok.ast tokay.tok.ast > tokay.diff) between both files:

38c38
<          range [start 23:15, end 23:18] => "AZ"
---
>          range [start 23:15, end 23:18]
40c40
<          range [start 23:19, end 23:22] => "az"
---
>          range [start 23:19, end 23:22]
73c73
<        range [start 38:6, end 38:9] => "07"
---
>        range [start 38:6, end 38:9]
81,83c81,83
<        range [start 42:6, end 42:9] => "09"
<        range [start 42:9, end 42:12] => "AF"
<        range [start 42:12, end 42:15] => "af"
---
>        range [start 42:6, end 42:9]
>        range [start 42:9, end 42:12]
>        range [start 42:12, end 42:15]
90c90
<      value_string [start 46:9, end 46:15] => ""
---
>      value_string [start 46:9, end 46:15] => "07"
93c93
<      value_string [start 47:9, end 47:15] => ""
---
>      value_string [start 47:9, end 47:15] => "08"
96c96
<      value_string [start 48:9, end 48:15] => ""
---
>      value_string [start 48:9, end 48:15] => "0c"
108c108
<      value_string [start 52:9, end 52:15] => ""
---
>      value_string [start 52:9, end 52:15] => "0b"
153c153
<        range [start 65:6, end 65:9] => "AZ"
---
>        range [start 65:6, end 65:9]
158,159c158,159
<         range [start 65:13, end 65:16] => "09"
<         range [start 65:16, end 65:19] => "AZ"
---
>         range [start 65:13, end 65:16]
>         range [start 65:16, end 65:19]
161c161
<         range [start 65:20, end 65:23] => "az"
---
>         range [start 65:20, end 65:23]
176c176
<        range [start 69:6, end 69:9] => "AZ"
---
>        range [start 69:6, end 69:9]
178c178
<        range [start 69:10, end 69:13] => "az"
---
>        range [start 69:10, end 69:13]
182,183c182,183
<         range [start 69:16, end 69:19] => "09"
<         range [start 69:19, end 69:22] => "AZ"
---
>         range [start 69:16, end 69:19]
>         range [start 69:19, end 69:22]
185c185
<         range [start 69:23, end 69:26] => "az"
---
>         range [start 69:23, end 69:26]
215c215
<      call [start 77:9, end 77:25]
---
>      call [start 77:9, end 77:26]
246c246
<      call [start 85:9, end 85:25]
---
>      call [start 85:9, end 85:26]
418c418
<      call [start 137:27, end 137:40]
---
>      call [start 137:27, end 137:96]
896,898c896
<        inline_sequence [start 256:13, end 256:23]
<         identifier [start 256:13, end 256:22] => "Attribute"
<        inline_sequence [start 256:25, end 256:34]
---
>        list [start 256:25, end 256:34]
1095,1097c1093
<        inline_sequence [start 299:23, end 299:27]
<         value_token_touch [start 299:23, end 299:26] => ">"
<        inline_sequence [start 299:29, end 299:32]
---
>        list [start 299:29, end 299:32]
1251,1253c1247
<        inline_sequence [start 324:23, end 324:27]
<         value_token_touch [start 324:23, end 324:26] => ">"
<        inline_sequence [start 324:29, end 324:32]
---
>        list [start 324:29, end 324:32]
1525,1529c1519
<       inline_sequence [start 389:27, end 389:48]
<        identifier [start 389:27, end 389:36] => "Statement"
<        op_mod_peek [start 389:37, end 389:47]
<         identifier [start 389:42, end 389:47] => "T_EOL"
<       inline_sequence [start 389:50, end 389:59]
---
>       list [start 389:50, end 389:59]
1557c1547
<        value_true [start 398:47, end 398:51] => "true"
---
>        identifier [start 398:47, end 398:51] => "true"

Before #66 can be enforced, these (few) differences need to be solved or defined to be ok.

phorward commented 2 years ago

Fixed by 9ab9926 which will soon be merged back here again.