Lexer issues

[kaby76]

The FortranLexer.g4 was copied from https://github.com/AkhilAkkapelli/Fortran2023Grammar/blob/553123a023f70e9a524e2a4036be128978834c42/Fortran2023Lexer.g4. But, it contains many problems.

Comments

The 2023 Spec clearly states:

The character “!” initiates a comment except where it appears within a character context. The comment extends to the end of the line. If the first nonblank character on a line is an “!”, the line is a comment line. Lines containing only blanks or containing no characters are also comment lines. Comments may appear anywhere in a program unit and may precede the first statement of a program unit or follow the last statement of a program unit. Comments have no effect on the interpretation of the program unit.

• The lexer grammar cannot have a block comment. There is no such thing in Fortran 2023!
• The spec states that blank lines or lines with just spaces are "comments". We should have lexer rules for empty line comments.
• Comments should not be "skip". There are many reasons why you don't do this. But, primarily, it is harder to annotate the parse tree for reverse engineering when it is not in the token stream.

INCLUDE

• It's required for include statements, but it is missing. (It is in the old Fortran90 grammar.)
• According to the Spec: An INCLUDE line is not a Fortran statement. However, the grammar needs rules for this, otherwise inputs may not parse. E.g., https://github.com/antlr/grammars-v4/blob/9d19a6d1a5a5819d809a7a940be39774675f3503/fortran/fortran90/examples/flang/test/f90_correct/src/ac02.f90#L7
• In the fortran90 grammar, this was part of the parser grammar. But, I'm now thinking that it should be completely in the lexer because it really should not be in the parser grammar, or it should be in a preprocessor.

=>

[ivan-pi]

In practice include is handled by the preprocessor (after the C-like preprocessor has been applied). See for instance here: https://flang.llvm.org/docs/Preprocessing.html#in-short-a-fortran-preprocessor-should-work-as-if

Saving files with a capitalized extension, e.g. .F or .F90, will automatically invoke the C-like Fortran preprocessor that responds to statements like #include, #define and others. On the other hand include always get processed, because it is a Fortran-specific mechanism.