removal of the implicit "filtering" feature during parse tree construction

[jurgenvinju]

This is a proposal to change the behavior of the Rascal parse function in ParseTree. Not enough for a complete amendment proposal in my view. But perhaps we can escalate this later if others deem this necessary.

Please help in this discussion @PaulKlint @tvdstorm @DavyLandman

• Currently when we use the: parse function a parser is generated from the type[Tree] grammar parameter. So first the grammar is reified from the current declaration scope (all relevant rules are collected) and then the grammar is passed to the parser generator.
• The grammar reification operator # is a static compile-time operator.
• All is fine with this. Grammars can be passed from one function to another at run-time, after they have been reified at compile time. This way there is no dependence on the declaration context of a grammar at parser generation time, or at parsing time.
• However, the generated parser also features a "filtering and normalization" option: after parsing when the tree is constructed, it is normalized using overloaded functions on the constructor names of the syntax rules and on the amb and appl constructors. The overloaded functions are collected from the calling context of the parse function. The filter statement also plays a big role ere.

This latter feature is highly problematic:

• it negates the clear separation of concerns and staging of the reified grammars
• the implementation is, to say the least, murky as the parse plugin deeply queries into the interpreters implementation details to recover the right overloaded functions to call.
• this ports very badly to the compiled runtime.

An optimal solution would be that the # operator produces also a reified set of normalizing functions which are applicable in the given compile-time scope, but that would require serializable functions to work, and some other features which Rascal does not have yet. Also it is questionable if these functions are logically belonging to a "reified grammar".

An intermediate solution is to add a keyword parameter to each parse function, which contains a set of references to normalization functions. That way the user can "reifiy" the normalizations functions and pass them to the parser explicitly. The same normalizing functionality would be obtained at a slight cost of changing all calls to parser functions:

parse(input, src, normalizers={amb, \ifStatement, \whileStatement})

• I suspect that existing code that uses this functionality is relatively easily adapted to this new style.
• this solves the problem and removes the need for the hacky way the parse built-in queries into the interpreter

Your thoughts pls.

[jurgenvinju]

Additional benefit: if parse functions do not need to call rascal functions then the IFunction which is generated by IRascalValueFactory.parser does not have to lock the interpreter anymore. This is beneficial for the code complexity of the parser function and also for parallel calls to the parser (which happens in Eclipse and the LSP a lot while typing).

[jurgenvinju]

reminder: we do need post-parse filtering during tree construction to bring tree construction for ambiguous-grammars-which are-disambiguated-by-post-parse-filtering down to O(n^3) as opposed to in O(n^p). (p the length of the longest production minus 1). Namely, we can apply disambiguations on the binarized SPPF level like this instead of on the non-binarized Rascal Tree representation.

[DavyLandman]

I'm fine with moving it to a parameter of parse.

The only place where I know it's used is in the standard libraries java8 grammar. It's used to disambiguate actually ambigu type-aliases in java, but also for some left-overs from the sdf source.

[jurgenvinju]

This was done. Filtering is no longer an implicit side-by-side effect but a parameter of the parse function.