Closed abrasat closed 5 years ago
Hi - yes, the token list is just a list of tokens that can be iterated over/processed using foreach
, Linq etc. - can you send more details/pseudocode if this doesn't help? Cheers!
Hi, Thx for the reply. For instance these would be some tokens
[Token(Category = "placeholder", Example = "%1")]
Placeholder1,
[Token(Category = "placeholder", Example = "%2")]
Placeholder2,
[Token(Category = "placeholder", Example = "%3")]
Placeholder3,
[Token(Category = "comment", Example = "//")]
Comment1,
[Token(Category = "comment", Example = "///")]
Comment2,
[Token(Category = "assignment", Example = ":=")]
Assign,
[Token(Example = "(")]
LeftParen,
[Token(Example = ")")]
RightParen,
[Token(Example = ",")]
Comma,
And these would be some strings to tokenize and parse
var str1 = "#this is a comment line";
var str2 = "var a := b";
var str3 = "printf("Hello %1")"
For instance, it should be checked if a string to parse contains tokens from the "placeholder" category
Thanks for the follow-up. No, category information isn't exposed right now - you'd need to look it up manually based on Token<T>.Kind
.
Ok, thanks
When iterating the tokens generated by the tokenizer, is it possible to check if any TokenAttribute (category, example or description) is set, and, if yes, to get their values ? It would be also interesting to be able to check directly if the token list contains any token from a certain category.