petersalex27 / yew

Yew: Strongly and Statically and Dependently Typed, Functional Language
MIT License
1 stars 0 forks source link

remove line number from token info, replace with start-end position #16

Closed petersalex27 closed 9 months ago

petersalex27 commented 9 months ago

refactor lexer so it uses start-end position (position encodes line number and char number) instead of line number, char on line start number, and char on same line end number

for example:

source := []string{
  "some line\n",     // line 1, chars 1-10
  "another line\n", // line 2, chars 1-13
}

becomes:

source := "some line\nanother line\n"
// line 1, index 0(incl.)-10(excl.); line 2, index 10(incl.)-23(excl.)
// then, can do binary search for position to find what range the position is in
lines := []int{/*line 1*/ 10, /*line 2*/ 23}