This PR extends the semantic highlighter with generation of tokens to explicitly indicate absence of highlighting. To avoid generating an excessive number of such tokens, the tokenizer on-the-fly identifies maximal consecutive regions of "absence of highlighting" (separated by newlines) and generates only a single token per region.
This PR extends the semantic highlighter with generation of tokens to explicitly indicate absence of highlighting. To avoid generating an excessive number of such tokens, the tokenizer on-the-fly identifies maximal consecutive regions of "absence of highlighting" (separated by newlines) and generates only a single token per region.