Add partial syntax, source maps, removed `Keyword` from AST and other CLI fixes

kaleidawave commented 5 months ago

Removed Keyword from AST
- Added a map store to parsing state which can store positions for keywords
- This is used in the LSP which gives documentation about language features (what does a keyword do)
- Uses binary search to find keywords
- Only under an option
- Updated synthesis for changes in structure
Fixes to context state across REPL entries
Simplified the generator macro interpolation mode

Added partial syntax

Added the ability for type annotations and expression to be missing in the source. This allows the following to be valid AST (under a parse option)

const x = ;
const y = 
const z = 2

Which should allow type checking with invalid source which is vital to the LSP

Source map bindings

This adds source map generation using the positions in the output and source

While possible a while ago. It needed this fix https://github.com/kaleidawave/source-map/pull/1

Added printing arguments across new lines for function calling

WIP other structures could benefit from this. Implementation feels like it has a bit of overhead. Indent could be better handled

Still to do

[x] experiment more with cursors (now markers and partial syntax)
[x] Finish the change in the generator macro
[x] add source map bindings in
[x] (maybe) add prettier printing for function calls etc?
[ ] try crush some fuzzing issues

kaleidawave commented 5 months ago

Hey @jasikpark you know about fuzzing! Looking at a recent naive run it is showing a mismatch

thread '<unnamed>' panicked at fuzz_targets/module_roundtrip_naive.rs:24:9:
input: `2
.A`
output1: `2.A;`

I was a little confused at first but I think input: ... is printing the input data. Is it possible to print output1 there instead? Thx in advance 🙏

kaleidawave commented 5 months ago

Really want to get ✅ on one of the fuzzing tests. While most of the time the errors are extreme edge cases, it would still be nice. Unfortunately they don't build on windows so having to do this trial and error remotely.

There also seems to be a regression in parsing performance introduced in this PR :(.

kaleidawave commented 5 months ago

Merging, will open an issue for the fuzzing tests

kaleidawave / ezno