Move INCLUDE resolution into the Lexer

[MarkusAmshove]

Currently copy codes are resolved within the parser, where a copy code is lexed and parsed when encountering an INCLUDE, passing all the copy code parameters to the lexer. The parsed statements from parsing the copy code are then appended after the INCLUDE node within the including source.

This works for a lot of cases, but does leave some edge cases behind and make them near impossible to solve.

A way to solve it would be to get the copy code resolution out of the parser and move it into the lexer. The lexer would encounter the INCLUDE keyword and then look for an identifier next and collect all the string literals that follow as copy code parameter. Then the lexer would call a new lexer for the copy code source and pass all parameter. The "nested" lexer returns a token list which is then appended instead of the INCLUDE identifier 'param'....

That should solve the following issues:

• 396

• 387

• ❗ Solve nested copy codes that pass parameters to another copy code, e.g. INCLUDE CC '&1'
• Any issues having DEFINE DATA be declared within a copy code (YFIBJMN0)

Implementation gotchas:

• The diagnostic position of nested lexemes needs to be put onto the INCLUDE or identifier of the copy code
• Diagnostics of the tokens that are replaced for parameters need to have a diagnostic position at the parameter token (can be done by passing the string literals as tokens instead of string values)

[Claes65]

This works for a lot of cases, but does leave some edge cases behind and make them near impossible to solve.

If you fully exploit the power of Copycodes, it's no longer just a corner case. So very happy to see this going into the project. 👍