Add ability to optionally create and use a custom PatternContext in order to override the default patterns that ship with the library.
Add ability to optionally capture Emoji tokens in the tokenizer.
Added some tests
I had the use case to use some different patterns than what shipped with the library. I also needed to capture Emoji tokens. I added new interfaces for this: PatternContext and EmojiExtractor. The DefaultPatternContext is used by default and has all of the patterns that originally shipped with the library. NoOpEmojiExtractor is used by default and will not capture Emojis.
I tested that the patch is backward compatible. Existing methods use defaults that ensure the tokens produced are the same as they are with the previous version of the library. I ran a test that processed 5,000 tweets with a fresh copy of master and then processed the same 5,000 tweets with this patch using all defaults. Both tests produced identical tokens.
Let me know if there are any problems or concerns with the patch and I will gladly address them. Thanks for the library, it has been very useful to me.
Hello,
This branch adds:
I had the use case to use some different patterns than what shipped with the library. I also needed to capture Emoji tokens. I added new interfaces for this: PatternContext and EmojiExtractor. The DefaultPatternContext is used by default and has all of the patterns that originally shipped with the library. NoOpEmojiExtractor is used by default and will not capture Emojis.
I tested that the patch is backward compatible. Existing methods use defaults that ensure the tokens produced are the same as they are with the previous version of the library. I ran a test that processed 5,000 tweets with a fresh copy of master and then processed the same 5,000 tweets with this patch using all defaults. Both tests produced identical tokens.
Let me know if there are any problems or concerns with the patch and I will gladly address them. Thanks for the library, it has been very useful to me.
Thanks, Steve