search

datalust / superpower

A C# parser construction toolkit with high-quality error reporting
Apache License 2.0
1.06k stars 98 forks source link

Tokenize<TKind>.Tokenize should accept TextSpan #85

Open sebastienros opened 5 years ago

sebastienros commented 5 years ago

I am pre-parsing some text to extract the parts I need to parse, and don't want to allocate more than necessary. For this reason it would be nice to be able to pass a TextSpan that I would extract from the original buffer instead of having to create a string out of it and pass it to the tokenizer.

sebastienros commented 5 years ago

Mitigation right now is to copy-paste the protected implementation in my custom tokenizer.

nblumhardt commented 5 years ago

That's interesting - thanks for the note 👍

nblumhardt commented 5 years ago

The challenge here is going to be that Tokenize(TextSpan) already exists in Tokenizer but has a different return type and semantics :-/ ... I'd guess we'll find some way around it, I'll give it some thought.