Add position information (byte offset) to lexemes

chewxy / lingo

package lingo provides the data structures and algorithms required for natural language processing

MIT License

151 stars 15 forks source link

Add position information (byte offset) to lexemes #18

Closed xeoncross closed 4 years ago

xeoncross commented 4 years ago

lexemes currently have row, column offset information. However, this requires scanning the whole source string to compute the position.

This PR adds a byte position offset.

xeoncross commented 4 years ago

Currently having an issue with the lexer tests not reporting the correct lex.Col values as well as correctly calculating the new .Position value. I could use some help.