search

crozone / FormatWith

String extensions for named parameterized string formatting.
MIT License
73 stars 13 forks source link

Vectorize the tokenizer #28

Closed crozone closed 3 years ago

crozone commented 3 years ago

Currently the tokenizer is a simple for loop over the input string. This is simple to implement and understand, but it leaves some performance on the table. Vectorization (eg SIMD) can be used to more quickly search the input string for noteworthy symbols (open and closed brackets).

MemoryExtensions.IndexOfAny() has been internally vectorized in .NET 5. If the tokenizer is rewritten to use IndexOfAny() to jump forwards to the next noteworthy token, it can jump over strides of text much more quickly.

crozone commented 3 years ago

Implemented in 918fa88e250cb80212e1159a48f559c791ab1a7d