The Tokenizer...TryUnundle local static method uses unsafe code and is very difficult to follow. While this was part of a PR that made significant perf improvements, it is not obvious that there is significant perf improvement vs. a character array.
Consider simplifying this code, or look for approach to clarify the intent and approach.
There is no rush on this.
It would be highly desirable to do this as part of, or after benchmarking efforts so we ensure we know if we backslide on perf improvements.
The
Tokenizer...TryUnundlelocal static method uses unsafe code and is very difficult to follow. While this was part of a PR that made significant perf improvements, it is not obvious that there is significant perf improvement vs. a character array.Consider simplifying this code, or look for approach to clarify the intent and approach.
There is no rush on this.
It would be highly desirable to do this as part of, or after benchmarking efforts so we ensure we know if we backslide on perf improvements.