Can't locate documentation: parsing non-char streams.

[katherinefluxinc]

Where can I find documentation about (of an example of) how to parse non-char tokens?

I have my own tokenizer that spits out tokens which are defined as:

  public record struct Token(TokenType TokenType, string Text, int Line, int Column)

I'd like to use Pidgin to parse these tokens. I've implemented an ITokenStream wrapping my tokenizer which, at least when tested in isolation, seems to function appropriately, but have been unable to locate documentation or examples elaborating on how I can write Parsers handling these non-char tokens.

It would be great if the documentation pointed me in the right direction to figure out how to do this.

Thanks very much in advance.

[benjamin-hodgson]

You can use the Parse(IEnumerable<TToken>) overload. Shouldn't need to implement ITokenStream except for advanced uses.

Hope this helps? Let me know if you have further questions.

[katherinefluxinc]

Thanks, I'll try that!

In the changelog, I had seen this item on the 3.0.0 release: "Published the (previously internal) TokenStream API. You can now write parsers which consume custom input streams."

This lead me to think that I would want to implement an ITokenStream providing my custom tokens, but I see now that while ITokenString itself is public, there are no public methods that accept ITokenStreams as parameters, and that the only method with an ITokenStream parameter is a private DoParse... this leaves me a little unsure what a user is meant to actually do with an ITokenStream they've written themself, but for now I'll proceed with trying to use the Parse(IEnumerable) method instead.

Thanks very much for the help!

[benjamin-hodgson]

My bad! I’d intended to make this method public but must’ve overlooked it. Will fix in a patch release. (Although I think most users won’t need to write their own streams in most circumstances.)

[benjamin-hodgson]

Shipped the missing method in 3.2.3 https://www.nuget.org/packages/Pidgin/3.2.3. Thanks!