Open zz1559152814 opened 4 years ago
MemCustomDictionary memCustomDictionary = new MemCustomDictionary(); FluentLexerBuilder builder = Lexers.coreBuilder(); builder.with(new CustomDictionaryPlugin(memCustomDictionary)); Lexer tokenizer = builder.build(); System.out.println(tokenizer); System.out.println(tokenizer.scan("欢迎来到松江临港科技城")); memCustomDictionary.addWord("临港科技城"); memCustomDictionary.rebuild(); System.out.println(tokenizer.scan("欢迎来到松江临港科技城"));
Output:
PipelineTokenizer BestPathAlgorithm = ViterbiBestPathAlgorithm CharNormalize = DefaultCharNormalize WordTermCollector = SentenceCollector WordSplitAlgorithm = CoreDictionarySplitAlgorithm,AtomSplitAlgorithm WordpathProcessor = CustomDictionaryProcessor 欢迎 来到 松江 临港 科技城 欢迎 来到 松江 临港科技城
Output: