Add Text.Earley.nextToken

[ocharles]

This new function lets production rules inspect the next token. Parsing decisions can't be made from this lookahead, so the grammar remains context free. One application of this new function is to extract location information from tokens - if the user has tokens with location information, we can use liftA2 addLocation nextToken r to run r, while also inspecting the first token r consumed. For example,

located
  :: Prod r Text (T.Located T.Token) Syntax
  -> Prod r Text (T.Located T.Token) Syntax
located p = do
  ~(T.Located x y _) <- nextToken
  syntax <- p
  return case syntax of
    At _ _ s -> At x y s
    s        -> At x y s

[ocharles]

Good idea! I'll try and throw something together.

[ocharles]

I'm afraid I'm unlikely to find the time to write a test. I had a look and I can't quite see where I'd put such a test. If you could help me with that, I'm probably able to write one.

[ollef]

The most natural place would probably be a new test file, perhaps looking at https://github.com/ollef/Earley/blob/master/tests/Optional.hs or some of the other files for inspiration.

[safinaskar]

@ocharles , your motivation is to have location info? There is no need to modify Earley library to do this. First, you need a lexer, which attaches location info to every token (for example, "lexer-applicative"), i. e. location of both begin of token and end of token. Then, you need to combine this info in Earley rules. http://hackage.haskell.org/package/language-lua2 is example of package, which does this

[ocharles]

@safinaskar yes, that's what the example in the original comment shows (note I have Located Tokens)

[ocharles]

I'm gonna go ahead and close this as I can't work out why I actually needed it! :laughing: