search

bzick / tokenizer

Tokenizer (lexer) for golang
MIT License
92 stars 6 forks source link

How to ignore _ in string? #3

Closed dviljoen closed 2 years ago

dviljoen commented 2 years ago

I'm trying to parse a URL or path-type string delimited by forward slashes. But it needs to be able to escape individual values that contain forward slashes with a backslash. I'm creating a tokenizer like this: tk := tokenizer.New() tk.DefineStringToken(1, "/", "/").SetEscapeSymbol(tokenizer.BackSlash) This works, except for values that contain underscores. It is splitting those into parts. How do I get the tokenizer to ignore underscores and not split on them?

dviljoen commented 2 years ago

And never mind. I found the AllowKeywordUnderscore() function.