Define a new version of the grammar to remove the need for `@IDENTIFIER`s

[misalcedo]

The following are the changes to the grammar:

• [x] #64
• [x] #65
• [ ] Add support for intervals.
• [ ] Map from the parser output to an AST
• [ ] implement TryFrom for Tokens instead of the method I use now.

[misalcedo]

Because of the way pest rules work, to define a parser I am basically writing a new recursive descent parser but with backtracking. I am going to explore the state machine approach before committing to PEST for the front-end. Pest does allow for fast experimentation, but it does not generate a syntax tree. Instead you have this pseudo tree that is the nested Pairs iterators.

I do like the new AST though. One of my decisions from this exercise is to make comparisons a pattern (i.e. refinement) for numbers and to just allow programs to have patterns instead of having the continuation entry point.

I need to rethink my entire pattern structure. Right now, patterns are expressions. However, without a boolean type they are closer to statements as they either throw an error, bind a variable, or filter which function definition to apply.

I have patterns as an expression in order to allow ergonomic currying like:

f(x) = g(y) = h(z) = x + y + z

The value of an equality pattern is to either throw an error or to bind the variable. However, I want the error to be a compile time error. That means that equality patterns are purely assignments. Patterns are not to be littered in code as that makes error handling proliferate through the code.

In order to tell a comparison apart from an assignment in semantic analysis, we need to ensure that all the variables are bound. If we are parsing for the interpreter, we allow the comparison otherwise we through a compile time error if all the variables are already bound.

Actually, this complication only arises because we want to have different behavior in the interpreter than a program. I am going to remove comparisons entirely and just make the interpreter be a program typed in line by line.

[misalcedo]

With comparisons not being allowed as expressions, that means we have only a binding expression and arithmetic expressions. Binding expressions allow nested binding expressions for currying. The open question is whether my example above should define only f in the outer scope (and define g and h in nested scopes) or to define each of the names along with f.

I don't think it makes sense to call g with calling f since x would not be bound. So, I am going to say that all binding operations define a new lexical scope to the right side (single or multi-line).

[misalcedo]

For bindings, The patterns on the left-hand side are either deconstructing or function definitions. A deconstruction pattern is:

• breaking an interval into its endpoints.
• separating a number into the whole and fractional portions.
• separating a tuple into its individual fields.

Since lists are just syntactic sugar on tuples I created #86 to track the work of adding the sugar.

Defining a function is a name with optional parameters. The parameters are also predicates that must all be true in order for a function to apply. Because of this, parameters have more expressive patterns than the expression pattern syntax.

[misalcedo]

As in Lox we can treat patterns as expressions. In the interpreter fully bound patterns are just evaluated as expressions.

Then to deal with assignments we parse an expression for the LHS, unpack it as a pattern, and return the assignment. This isn't necessarily pretty but helps to handle assignment in the parser without backtracking.

Semantic analysis will handle cases where a pattern looks to the parser like a regular expression but the name is not yet defined.

[misalcedo]

For the syntax tree, I need to add references back to the source. Each node in the tree must have a start location and string or str for the lexeme.

That way errors can print a lexeme's location.

Goals:

• [ ] tree should be editable my making all fields public for mutable references.
• [ ] tree should be printable to allow reconstructing the exact same source file if unchanged (preserve blank space).
• [ ] tree should support owned and borrowed input

Non-Goals:

• [ ] tree does not need to encode operator precedence or associativity
• [ ] Space does not need to be preserved within changed portions.

Originally posted by @misalcedo in https://github.com/misalcedo/tortuga/issues/77#issuecomment-1016691562

[misalcedo]

The goal now is to make patterns an expression. Semantic analysis will error if a pattern is used outside of an assignment expression, but future grammar could use grammars for supporting reusable patterns of structures.

For example:

@Point = { x, y }
p = @Point({1, 2})
{ p:x, p:y }

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. You may re-open the issue if it is still relevant.