search

antlr / grammars-v4

Grammars written for ANTLR v4; expectation that the grammars are free of actions.
MIT License
10.2k stars 3.71k forks source link

Malformed start rules in grammars #2405

Closed kaby76 closed 2 years ago

kaby76 commented 2 years ago

I was reading on StackOverflow this question on identifying a start rule for a grammar. The answer, from @bkiers , was to look for the rule that contains an EOF.

So, I decided to see what grammars in this repo contain a rule with an EOF using this script:

for i in `find . -name 'pom.xml' | sed 's#/pom.xml##' | sort -u`
do
    pushd $i > /dev/null 2>&1
    shopt -s nullglob
    compgen -W *.g4 > /dev/null 2>&1
    case $? in
        0)
            echo ""
            echo *.g4
            trparse *.g4 | trxgrep ' //parserRuleSpec[*//TOKEN_REF/text()="EOF"]/RULE_REF' | trtext
            ;;
        1)
            echo ""
            echo *.g4
            trparse *.g4 | trxgrep ' //parserRuleSpec[*//TOKEN_REF/text()="EOF"]/RULE_REF' | trtext
            ;;
        2)  ;;
    esac
    popd > /dev/null 2>&1
done

(Note, I had to fix a bug in Trash to get this to work.)

Surprisingly, there are many grammars that do not contain a start rule with EOF, e.g., acme, agc, alef, asm/*, asn, bcpl, calculator, dot, fasta, http, json, pascal, turing, xpath (not inclusive).

Augmenting the start rule with the EOF symbol at the end is important because it forces the parser to consume the entire file. See https://github.com/antlr/antlr4/issues/118. To illustrate the problem, modify the calculator grammar with a rule "file: expression;", and try running "parser.file();" on input "1 2". It succeeds, although it really should fail.

Alternatively, I could modify the templates for testing to check the state of the parse tree input against the lexer token stream to see whether the input is fully consumed by the parser.

kaby76 commented 2 years ago

Following up on what we learned in the Scala grammar, I wrote a new script to go over every grammar and find grammars without an EOF terminated start rule. The script then generates a parser, modifies the start rule of the grammar to terminate with an EOF, then redoes the tests. The change forces the parser to consume the entire input in each test.

Results

Grammars that are missing EOF terminated start rules

Missing EOF in start rule acmeCompUnit for grammar ./acme
acmeCompUnit : (acmeImportDeclaration)* (acmeSystemDeclaration | acmeFamilyDeclaration | acmeDesignDeclaration)+ ;

Missing EOF in start rule prog for grammar ./agc
prog : line+ ;

Missing EOF in start rule program for grammar ./alef
program : decllist? ;

Missing EOF in start rule alloyModule for grammar ./alloy
alloyModule : moduleDecl? import_* paragraph+ ;

Missing EOF in start rule prog for grammar ./alpaca
prog : defns ('.' | 'begin') ;

Missing EOF in start rule script for grammar ./angelscript
script : (import_ | enum_ | typdef | class_ | mixin_ | interface_ | funcdef | virtprop | var_ | func_ | namespace | ';')+ ;

Missing EOF in start rule record for grammar ./apt
record: commented=commenterR? rType=TypeR WSS options=optionsR? uri=uriR WSS distribution=wordWithDash components=componentsR WSS?;

Missing EOF in start rule module for grammar ./argus
module : equate* equates_ | equate* guardian | equate* procedure | equate* iterator | equate* cluster ;

Missing EOF in start rule equation for grammar ./arithmetic
equation : expression relop expression ;

Missing EOF in start rule prog for grammar ./asm/asm6502
prog : (line? EOL) + ;

Missing EOF in start rule prog for grammar ./asm/asm8080
prog : (line? EOL) + ;

Missing EOF in start rule prog for grammar ./asm/asm8086
prog : (line ('!' line)* EOL)* ;

Missing EOF in start rule prog for grammar ./asm/asmMASM
prog : (line EOL)* ;

Missing EOF in start rule prog for grammar ./asm/asmZ80
prog : (line? EOL) + ;

Missing EOF in start rule compilationUnit for grammar ./asm/masm
compilationUnit : (segments | directive_exp1)* 'end' Identifier ;

Missing EOF in start rule moduleDefinition for grammar ./asn/asn
moduleDefinition : IDENTIFIER (L_BRACE (IDENTIFIER L_PARAN NUMBER R_PARAN)* R_BRACE)? DEFINITIONS_LITERAL tagDefault extensionDefault ASSIGN_OP BEGIN_LITERAL moduleBody END_LITERAL ;

Missing EOF in start rule moduleDefinition for grammar ./asn/asn_3gpp
moduleDefinition : IDENTIFIER (L_BRACE (IDENTIFIER L_PARAN NUMBER R_PARAN)* R_BRACE)? DEFINITIONS_LITERAL tagDefault extensionDefault ASSIGN_OP BEGIN_LITERAL moduleBody END_LITERAL ;

Missing EOF in start rule unit for grammar ./atl
unit : module | library_ | query ;

Missing EOF in start rule program for grammar ./b
program : definition* ;

Missing EOF in start rule program for grammar ./bcpl
program : (declaration_part | directive) ;

Missing EOF in start rule file_ for grammar ./brainflak
file_ : statement+ ;

Missing EOF in start rule file_ for grammar ./brainfuck
file_ : statement* ;

Missing EOF in start rule equation for grammar ./calculator
equation : expression relop expression ;

Missing EOF in start rule program for grammar ./callable
program : line (EOL+ line)* ;

Missing EOF in start rule expr for grammar ./cayenne
expr: '(' varid '::' type_ ')' '->' expr | '\\' '(' varid '::' type_ ')' '->' expr | expr expr | 'data' (conid (type_)* '|')* | conid '@' type_ | 'case' varid 'of' arm* '::' type_ | 'sig' sign* | 'struct' defn* | expr '.' lblid | id_ | '#';

Missing EOF in start rule log for grammar ./clf
log : (line? EOL) + line? ;

Missing EOF in start rule termseq for grammar ./clif
termseq : (term | SEQMARK)* ;

Missing EOF in start rule module for grammar ./clu
module : equate* (procedure | iterator | cluster) ;

Missing EOF in start rule cookie for grammar ./cookie
cookie : av_pairs* ;

Missing EOF in start rule program for grammar ./cool
program : programBlocks ;

Missing EOF in start rule root for grammar ./cql3
CqlParser.g4:root : cqls? MINUSMINUS? eof ;

Missing EOF in start rule document for grammar ./creole
document : (line? CR)* ;

Missing EOF in start rule stylesheet for grammar ./css3
stylesheet : ws ( charset ( Comment | Space | Cdo | Cdc )* )* ( imports ( Comment | Space | Cdo | Cdc )* )* ( namespace_ ( Comment | Space | Cdo | Cdc )* )* ( nestedStatement ( Comment | Space | Cdo | Cdc )* )* ;

Missing EOF in start rule csvFile for grammar ./csv
csvFile: hdr row+ ;

Missing EOF in start rule proposition for grammar ./ctl
proposition : CTL_DOWNTACK | CTL_UPTACK | ATOMIC | CTL_INEVITABLE (CTL_NEXT | CTL_FINALLY | CTL_GLOBALLY) proposition | CTL_EXISTS (CTL_NEXT | CTL_FINALLY | CTL_GLOBALLY) proposition | CTL_INEVITABLE '[' proposition CTL_UNTIL proposition ']' | CTL_EXISTS '[' proposition CTL_UNTIL proposition ']' | '(' proposition ')' | proposition (CTL_AND | CTL_OR | CTL_RIGHTWARDS_DOUBLE_ARROW | CTL_LEFT_RIGHT_DOUBLE_ARROW) proposition | CTL_NOT proposition ;

Missing EOF in start rule databank for grammar ./databank
databank : EOL* (datedseries | undatedseries) sample + ;

Missing EOF in start rule konservierung for grammar ./dcm
konservierung : ( '\n' )* 'KONSERVIERUNG_FORMAT 2.0' ( '\n' )+ kons_kopf kons_rumpf ;

Missing EOF in start rule module for grammar ./dgol
module : (usedeclaration | NL)* (subroutinedefinition | NL)* (programdefinition | librarydefinition) NL* ;

Missing EOF in start rule notation for grammar ./dice
DiceNotationParser.g4:notation : dice | number | addOp ;

Missing EOF in start rule graph for grammar ./dot
graph : STRICT? ( GRAPH | DIGRAPH ) id_? '{' stmt_list '}' ;

Missing EOF in start rule goal for grammar ./edif300
goal : edif;

Missing EOF in start rule sequence for grammar ./fasta
sequence : section+ ;

Missing EOF in start rule fen for grammar ./fen
fen : placement ' ' color ' ' castling ' ' enpassant ' ' halfmoveclock ' ' fullmoveclock ;

Missing EOF in start rule schema for grammar ./flatbuffers
schema : include_* ( namespace_decl | type_decl | enum_decl | union_decl | root_decl | file_extension_decl | file_identifier_decl | attribute_decl | rpc_decl | object_ )* ;

Missing EOF in start rule prog for grammar ./focal
prog : statement+ ;

Missing EOF in start rule program for grammar ./fortran77
Fortran77Parser.g4:program : commentStatement* (executableUnit commentStatement*)+ EOL* ;

Missing EOF in start rule fusionTablesSql for grammar ./fusion-tables
fusionTablesSql : sql_stmt FusionTablesSql.g4 pom.xml README.md ;

Missing EOF in start rule gedcom for grammar ./gedcom
gedcom : line+ ;

Missing EOF in start rule document for grammar ./gff3
document : HEADER line+ ;

Missing EOF in start rule graph for grammar ./gml
graph : kv + ;

Missing EOF in start rule document for grammar ./graphql
document: definition+;

Missing EOF in start rule dgs for grammar ./graphstream-dgs
DGSParser.g4:dgs : header ( event | COMMENT | EOL )* ;

Missing EOF in start rule prog for grammar ./guido
prog : (segment +) | sequencelist ;

Missing EOF in start rule tab for grammar ./guitartab
tab : string+ ;

Missing EOF in start rule htmlDocument for grammar ./html
HTMLParser.g4:htmlDocument : scriptletOrSeaWs* XML? scriptletOrSeaWs* DTD? scriptletOrSeaWs* htmlElements* ;

Missing EOF in start rule http_message for grammar ./http
http_message: start_line (header_field CRLF)* CRLF //message_body ;

Missing EOF in start rule prog for grammar ./icon
prog : declaration | (declaration prog) ;

Missing EOF in start rule specification for grammar ./idl
specification : import_decl* definition + ;

Missing EOF in start rule inf for grammar ./inf
inf : (section | EOL)* ;

Missing EOF in start rule conditionalRule for grammar ./infosapient
conditionalRule : IDENTIFIER premise consequent ';' ;

Missing EOF in start rule program for grammar ./janus
program : (IDENT ('[' NUM ']')?)* ('PROCEDURE' IDENT statements)* ;

Missing EOF in start rule compilationUnit for grammar ./java/java9
Java9Parser.g4:compilationUnit : ordinaryCompilation | modularCompilation ;

Missing EOF in start rule prog for grammar ./joss
prog : statement+ ;

Missing EOF in start rule ql_statement for grammar ./jpa
ql_statement : select_statement | update_statement | delete_statement ;

Missing EOF in start rule json for grammar ./json
json : value ;

Missing EOF in start rule karel for grammar ./karel
karel : 'BEGINNING-OF-PROGRAM' definition* 'BEGINNING-OF-EXECUTION' statement* 'END-OF-EXECUTION' 'END-OF-PROGRAM' ;

Missing EOF in start rule expression for grammar ./lambda
expression : VARIABLE | function_ | application ;

Missing EOF in start rule start for grammar ./lark
LarkParser.g4:start: (item? NL)* item? ;

Missing EOF in start rule stylesheet for grammar ./less
LessParser.g4:stylesheet : statement* ;

Missing EOF in start rule program for grammar ./limbo
program : 'implement' IDENTIFIER ';' top_declaration_sequence ;

Missing EOF in start rule program for grammar ./lisa
program: declaration_block program_block;

Missing EOF in start rule prog for grammar ./logo/logo
prog : (line? EOL) + line? ;

Missing EOF in start rule program for grammar ./lolcode
program : 'HAI' code_block 'KTHXBYE'? ;

Missing EOF in start rule proposition for grammar ./ltl
proposition : 'true' | 'false' | ATOMIC | '(' proposition ')' | proposition (LTL_AND | LTL_OR | LTL_RIGHTWARDS_SINGLE_ARROW) proposition | LTL_NOT proposition | (LTL_GLOBALLY | LTL_FINALLY | LTL_NEXT) proposition | proposition (LTL_UNTIL | LTL_WEAK | LTL_RELEASE) proposition ;

Missing EOF in start rule statement for grammar ./matlab
statement : global_statement | clear_statement | assignment_statement | expression_statement | selection_statement | iteration_statement | jump_statement ;

Missing EOF in start rule command_line for grammar ./memcached_protocol
command_line : (storage_command | retrieval_command | delete_command | increment_command | decrement_command | statistics_command | flush_command | version_command | verbosity_command | quit_command)+ ;

Missing EOF in start rule database for grammar ./metamath
database : outermostscopestmt* ;

Missing EOF in start rule uom for grammar ./metric
uom : measure (('*' | '/') measure)* ;

Missing EOF in start rule program for grammar ./microc
program : statement+ ;

Missing EOF in start rule stored_definition for grammar ./modelica
stored_definition : ('within' (name)? ';')* (('final')? class_definition ';')* ;

Missing EOF in start rule compilationUnit for grammar ./modula2pim4
compilationUnit : definitionModule | IMPLEMENTATION? programModule ;

Missing EOF in start rule molecule for grammar ./molecule
molecule : part_ ('ú' part_)* ;

Missing EOF in start rule prog for grammar ./moo
prog : declaration + ;

Missing EOF in start rule morsecode for grammar ./morsecode
morsecode : letter (SPACE letter)+ ;

Missing EOF in start rule db for grammar ./muddb
db : room* END ;

Missing EOF in start rule program for grammar ./mumps
program : line + eof ;

Missing EOF in start rule prog for grammar ./muparser
prog : expr ('\n' expr)* # progExpr ;

Missing EOF in start rule exp for grammar ./nanofuck
exp : '*' | '{' exp? '}' | exp exp ;

Missing EOF in start rule tree_ for grammar ./newick
tree_ : (rootLeaf ';') | (rootInternal ';') | (branch ';') ;

Missing EOF in start rule module for grammar ./oberon
module : MODULE ident ';' importList? declarationSequence (BEGIN statementSequence)? END ident '.' ;

Missing EOF in start rule oncrpcv2Specification for grammar ./oncrpc
oncrpcv2Specification : (xdrSpecification | programDef)*;

Missing EOF in start rule program for grammar ./orwell
program : decl+ ;

Missing EOF in start rule prog for grammar ./p
prog : symbol+ ;

Missing EOF in start rule parkingSigns for grammar ./parkingsign
parkingSigns : parkingSign* ;

Missing EOF in start rule program for grammar ./pascal
program : programHeading (INTERFACE)? block DOT ;

Missing EOF in start rule pddlDoc for grammar ./pddl
pddlDoc : domain | problem ;

Missing EOF in start rule game for grammar ./pdn
game : tags moves ;

Missing EOF in start rule program for grammar ./peoplecode
program : stmtList ;

Missing EOF in start rule program for grammar ./pike
program : definition* ;

Missing EOF in start rule program for grammar ./pl0
program : block '.' ;

Missing EOF in start rule program for grammar ./plucid
program : expression ;

Missing EOF in start rule ply for grammar ./ply
ply : header vertices? faces? ;

Missing EOF in start rule commandlist for grammar ./pmmn
commandlist : command | commandlist command ;

Missing EOF in start rule postalcode for grammar ./postalcode
postalcode : LETTER DIGIT LETTER DIGIT LETTER DIGIT ;

Missing EOF in start rule document for grammar ./powerquery
PowerQueryParser.g4:document: section_document | expression_document;

Missing EOF in start rule proposition for grammar ./propcalc
proposition : expression THEREFORE expression ;

Missing EOF in start rule propertiesFile for grammar ./properties
propertiesFile : row+ ;

Missing EOF in start rule proto for grammar ./protobuf2
proto : syntax ( importStatement | packageStatement | optionStatement | topLevelDef | emptyStatement_ )* ;

Missing EOF in start rule proto for grammar ./protobuf3
proto : syntax ( importStatement | packageStatement | optionStatement | topLevelDef | emptyStatement_ )* ;

Missing EOF in start rule document for grammar ./prov-n
document : DOCUMENT (namespaceDeclarations)? (expression)* (bundle (bundle)*)? ENDDOCUMENT ;

Missing EOF in start rule map_ for grammar ./quakemap
map_ : entity* ;

Missing EOF in start rule program for grammar ./racket-bsl
program : defOrExpr+ ;

Missing EOF in start rule program for grammar ./racket-isl
program : defOrExpr+ ;

Missing EOF in start rule file_ for grammar ./redcode
file_ : line + ;

Missing EOF in start rule program for grammar ./refal
program : f_definition (';'? program)? | external_decl ';' program | program external_decl ';' ;

Missing EOF in start rule domain for grammar ./rfc1035
domain : subdomain | ' ' ;

Missing EOF in start rule filter_ for grammar ./rfc1960
filter_ : '(' filtercomp ')' ;

Missing EOF in start rule frame for grammar ./rfc3080
frame : data ;

Missing EOF in start rule date_time for grammar ./rfc822/rfc822-datetime
date_time : (day ',')? date time ;

Missing EOF in start rule emailaddress for grammar ./rfc822/rfc822-emailaddress
emailaddress : mailbox | group ;

Missing EOF in start rule program for grammar ./robotwars
program : line + ;

Missing EOF in start rule expression for grammar ./romannumerals
expression : thousands ;

Missing EOF in start rule expression for grammar ./rpn
expression : signedAtom term* ;

Missing EOF in start rule prog for grammar ./ruby
prog : expression_list;

Missing EOF in start rule prog for grammar ./scotty
prog : program_lines ;

Missing EOF in start rule stylesheet for grammar ./scss
ScssParser.g4:stylesheet : statement* ;

Missing EOF in start rule collection for grammar ./sgf
collection : gameTree+;

Missing EOF in start rule prog for grammar ./sharc
SHARCParser.g4:prog : ( statement SEMICOLON )+ ;

Missing EOF in start rule sickbay for grammar ./sickbay
sickbay : line* ;

Missing EOF in start rule start for grammar ./sieve
start : commands ;

Missing EOF in start rule smiles for grammar ./smiles
smiles: chain terminator?;

Missing EOF in start rule prog for grammar ./snobol
prog : lin + ;

Missing EOF in start rule program for grammar ./snowball
program : pom.xml ;

Missing EOF in start rule statement for grammar ./sql/hive/v2
HiveParser.g4:statement : explainStatement | execStatement ;

Missing EOF in start rule statement for grammar ./sql/hive/v3
HiveParser.g4:statement : explainStatement | execStatement ;

Missing EOF in start rule content for grammar ./stellaris
content: expr+ ;

Missing EOF in start rule template_ for grammar ./stringtemplate
STGParser.g4:template_ : ( AT ID DOT ID LPAREN RPAREN | ID LPAREN formalArgs? RPAREN ) TMPL_ASSIGN ( STRING // "..." | BIGSTRING // <<...>> | BIGSTRING_NO_NL // <%...%> ) | ID TMPL_ASSIGN ID // alias one template to another ;

Missing EOF in start rule top_level for grammar ./suokif
top_level : sentence* ;

Missing EOF in start rule segmentheader for grammar ./tcpheader
segmentheader : sourceport destinationport sequencenumber acknumber flags windowsize checksum urgent ;

Missing EOF in start rule number for grammar ./telephone
number : '+1'? '+'? variation ;

Missing EOF in start rule file_ for grammar ./terraform
file_ : (local | module | output | provider | variable | data | resource | terraform)+ ;

Missing EOF in start rule program for grammar ./tiny
program : 'BEGIN' stmt_list 'END' ;

Missing EOF in start rule program for grammar ./tinybasic
program : line* ;

Missing EOF in start rule program for grammar ./tinyc
program : statement + ;

Missing EOF in start rule prog for grammar ./tinymud
prog : line + EOL* ;

Missing EOF in start rule proposition for grammar ./tl
proposition : | TL_UPTACK | ATOMIC | TL_NOT proposition | proposition TL_OR proposition | (TL_ALWAYS | TL_WAS) proposition | '(' proposition ')' ;

Missing EOF in start rule tnsnames for grammar ./tnsnames
tnsnamesParser.g4:tnsnames : (tns_entry | ifile | lsnr_entry)* ;

Missing EOF in start rule equation for grammar ./tnt
equation : expression '=' expression ;

Missing EOF in start rule tsvFile for grammar ./tsv
tsvFile : hdr row* ;

Missing EOF in start rule program for grammar ./turing
program : declarationOrStatementInMainProgram+ ;

Missing EOF in start rule turtleDoc for grammar ./turtle
turtleDoc : statement* ;

Missing EOF in start rule turtleDoc for grammar ./turtle-doc
turtleDoc : statement* ;

Missing EOF in start rule codepoint for grammar ./unicode/unicode16
codepoint: CLASSIFY___ // Error | CLASSIFY_Cc // Control | CLASSIFY_Cf // Format | CLASSIFY_Cn // Unassigned | CLASSIFY_Co // Private_Use | CLASSIFY_Cs // Surrogate | CLASSIFY_Ll // Lowercase_Letter | CLASSIFY_Lm // Modifier_Letter | CLASSIFY_Lo // Other_Letter | CLASSIFY_Lt // Titlecase_Letter | CLASSIFY_Lu // Uppercase_Letter | CLASSIFY_Mc // Spacing_Mark | CLASSIFY_Me // Enclosing_Mark | CLASSIFY_Mn // Nonspacing_Mark | CLASSIFY_Nd // Decimal_Number | CLASSIFY_Nl // Letter_Number | CLASSIFY_No // Other_Number | CLASSIFY_Pc // Connector_Punctuation | CLASSIFY_Pd // Dash_Punctuation | CLASSIFY_Pe // Close_Punctuation | CLASSIFY_Pf // Final_Punctuation | CLASSIFY_Pi // Initial_Punctuation | CLASSIFY_Po // Other_Punctuation | CLASSIFY_Ps // Open_Punctuation | CLASSIFY_Sc // Currency_Symbol | CLASSIFY_Sk // Modifier_Symbol | CLASSIFY_Sm // Math_Symbol | CLASSIFY_So // Other_Symbol | CLASSIFY_Zl // Line_Separator | CLASSIFY_Zp // Paragraph_Separator | CLASSIFY_Zs // Space_Separator ;

Missing EOF in start rule searchCrit for grammar ./upnp
searchCrit : searchExp | ASTERISK ;

Missing EOF in start rule prog for grammar ./useragent
prog : (product comment?) + ;

Missing EOF in start rule sourceFile for grammar ./v
sourceFile : (moduleClause eos)? ( importDecl eos )* ( topLevelDecl eos )* ;

Missing EOF in start rule source_text for grammar ./verilog/systemverilog
SystemVerilogPreParser.g4:source_text : compiler_directive* ;

Missing EOF in start rule source_text for grammar ./verilog/verilog
VerilogPreParser.g4:source_text : compiler_directive* ;

Missing EOF in start rule vmf for grammar ./vmf
vmf : keyvalue+ ;

Missing EOF in start rule module_ for grammar ./wat
WatParser.g4:module_ : LPAR MODULE VAR? module_field* RPAR ;

Missing EOF in start rule start for grammar ./wavefront
start : ( NL* statement ) ( NL+ statement )* NL* ;

Missing EOF in start rule geometry for grammar ./wkt
geometry : (polygonGeometry | lineStringGeometry | pointGeometry | compoundCurveGeometry | curvePolygonGeometry | multiSurfaceGeometry | multiCurveGeometry | multiPointGeometry | multiLineStringGeometry | multiPolygonGeometry | circularStringGeometry | multiPolyhedralSurfaceGeometry | multiTinGeometry | geometryCollection) ;

Missing EOF in start rule document for grammar ./xml
XMLParser.g4:document : prolog? misc* element misc*;

Missing EOF in start rule main for grammar ./xpath/xpath1
main : expr ;

Grammars that failed after correcting with an EOF terminated start rule

agc
asm/asm8086
asn/asn
bcpl
clif
databank
html
lambda
lark
metamath
muddb
mumps
nanofuck
pdn
rfc1960
sickbay
sql/hive/v2
tcpheader
turing
wavefront
wkt

The results show that there are many grammars that do not have EOF terminated start rules. It also shows that some of these grammars have not been tested so as to consume the full input.

It's likely there are hidden bugs in these grammars.

#!/bin/sh

test="1"
files=`find . -name pom.xml | grep -v Generated`
refined_list=`grep -l -i -e entrypoint $files`
for i in $refined_list
do
    dir=${i%/*}
    grammars=`trxml2 $i | grep -i 'includes/include=' | sed 's%.*/include=\(.*\)$%\1%'`
    sourcedir=`trxml2 $i | grep -i 'configuration/sourceDirectory=' | sed 's%.*/sourceDirectory=\(.*\)$%\1%' | sed 's/[$][{]basedir[}]//' | sed 's%^/%%g'`
    fixedgrammars=""
    for g in $grammars
    do
        if [[ $sourcedir == "" ]]
        then
            fixedgrammars="$fixedgrammars $g"
        else
            fixedgrammars="$fixedgrammars $sourcedir/$g"
        fi
    done
    entry=`trxml2 $i | grep -i 'configuration/entryPoint=' | sed 's%.*/entryPoint=\(.*\)$%\1%'`
    pushd $dir > /dev/null 2>&1
    for e in $entry
    do
        rule=`trparse -t antlr4 $fixedgrammars | trxgrep ' //parserRuleSpec[RULE_REF/text()="'$e'" and not(*//TOKEN_REF/text()="EOF")]' | trtext`
        if [[ $rule != "" ]]
        then
            echo Missing EOF in start rule $entry for grammar $dir
            echo $rule
            echo ""
            if [[ $test != "" ]]
            then
                rm -rf Generated
                trgen
                cd Generated
                echo $e
                trparse -t antlr4 *.g4 | trinsert " //parserRuleSpec[RULE_REF/text()='"$e"']/SEMI" "EOF" | trsponge -c
                make
                make test
                cd ..
                rm -rf Generated
            fi
        fi
    done

    popd > /dev/null 2>&1
done