Add trivia to tokens and parse runner.

[patrickhuber]

Text in the input that is not used in the parse but is important for refactoring tools is currently thrown away with ignore rules.

• Add a new class called trivia that captures the value of this text and it's position in the input.
• Trivia is associated with the tokens and each token can have leading and following trivia collections.
• Trivia is captured similar to how ignore lexer rules work, but instead of discarding the data, it will be cached. When a token is recognized, the trivia will be added as leading trivia to the token before it is added to the parser.
• May need to track line endings for trailing trivia. This may also require caching the previous token in the parse tunner state.
• Add a new directive to the grammar text for trivia ex: trivia: /\w+/ ; trivia: whitespace ;

[patrickhuber]

Some Requirements:

Grammar

Given Grammar:

RepeatingWord = 
    word RepeatingWord
    | word ;
word ~ /[\w]+/ ;
whitespace ~ /[\s]+/;
start: RepeatingWord ;
trivia: whitespace ;

Test 1

When Input

[space][space]aa[space]aa[space][space]

Then

first token leading trivia is [space][space]
first token following trivia is Empty
second token leading trivia is [space]
second token folliwing trivia is [space][space]

Test 2

When Input

aa[space]aa

Then

first token leading trivia is null
first token following trivia is null
second token leading trivia is [space]
second token following trivia is null

Test 3

When Input

aa[space][carriage return][line feed]aa[space]

Then

first token leading trivia is null
first token following trivia is [space][carriage return][line feed]
second token leading trivia is [space]
second token following trivia is [space]

Test 4

When Input

aa[space][carriage return][line feed]aa[space][carriage return][line feed][carriage return][line feed]

Then

first token leading trivia is null
first token following trivia is [space][carriage return][line feed]
second token leading trivia is [space]
second token following trivia is [space][carriage return][line feed][carriage return][line feed]

[patrickhuber]

@ArsenShnurkov please review the pull request #90 that implements trivia.