By storing the token data in a separate Scanner field and having Token be merely the token type, we can avoid a decent amount of copying when tokens are passed around. This leads to considerable speedups for the TokenReader and Reader benchmarks (the Scanner benchmark is slightly slower, but that probably has more to do with how that particular benchmark is written, since the token data was previously discarded).
By storing the token data in a separate
Scanner
field and havingToken
be merely the token type, we can avoid a decent amount of copying when tokens are passed around. This leads to considerable speedups for theTokenReader
andReader
benchmarks (theScanner
benchmark is slightly slower, but that probably has more to do with how that particular benchmark is written, since the token data was previously discarded).