Closed xzyfer closed 9 years ago
There are some cases where we have uses more specific sub-tokenizers i.e quoted string. In these cases we're not passing along the current position context so those tokenizers assume it's the start of the file.
There are some cases where we have uses more specific sub-tokenizers i.e quoted string. In these cases we're not passing along the current position context so those tokenizers assume it's the start of the file.