type Token

[isomorphisms]

You set

type Token struct {
    Type        int // the token type
    Value       interface{} // a value associate with the token
    Lexeme      []byte // the string that was matched
    TC          int // the index (text counter) in the string
    StartLine   int
    StartColumn int
    EndLine     int
    EndColumn   int
}

Are EndLines ever "above" StartLines? If not, shouldn't you use a uint type for the forward column and row offset and make EndLine and EndColumn derivatives of that?

[timtadh]

You could do that. If you need to do that you can supply your own token type. The provided Token struct is merely a usable example (fit for many but not all purposes). My purpose with this particular definition is simplicity -- representing end locations as a derivative of start locations is not simple. Also, end columns may be smaller than start columns for multi-line tokens.