While investigating #159 i found some unneeded code.
Instead of first splitting, joining, copying (or rather: constructing a new string from the old ones char array that is cast to a list) and then splitting again, the the first split operation is returned directly. Also the loops to add spaces around the tokens got moved into its own method.
For a few characters more ParseSurrounded, ParseSeperateTokens and the lambda in ParseTokens could return IEnumerator<...> and the toArray calls could be removed.
This would make these functions truely lazy and maybe give a small performance boost on the tokenizing step.
While investigating #159 i found some unneeded code. Instead of first splitting, joining, copying (or rather: constructing a new string from the old ones char array that is cast to a list) and then splitting again, the the first split operation is returned directly. Also the loops to add spaces around the tokens got moved into its own method.
For a few characters more
ParseSurrounded,ParseSeperateTokensand the lambda inParseTokenscould returnIEnumerator<...>and thetoArraycalls could be removed. This would make these functions truely lazy and maybe give a small performance boost on the tokenizing step.