Allow to skip the error

[omegascorp]

With this small change you don't have to throw an error in case of unknown term.

Example of usage (without the change it will be an infinite loop):

for token, err, eof = scanner.Next(); !eof; token, err, eof = scanner.Next() {
    log.Println(token)
}

[timtadh]

You don't need to patch the library in order to skip an error as you did your patch.

The TC variable (text counter) is exposed through the api to allow you to reset the Scanner's position from the for loop eg.

for tok, err, eos := scanner.Next(); !eos; tok, err, eos = scanner.Next() {
    if err != nil {
        if ui, is := err.(*machines.UnconsumedInput); is {
            // skip the error by resetting the scanner.TC
            scanner.TC = ui.FailTC
            continue
        }
        return err
    }
    // your token processing here
}

I need to work on the documentation to expose how to do some of this stuff. If this works (or doesn't) for you please let me know.

Edit

I should add, you need to check the status of the err variable every iteration of the loop. That way, if there is a bad token you can just drop out of the loop.

[omegascorp]

Thank you, looks good. And yeah, it is hard to figure out. I will close pull request.