set the token size limit and align it with lucene

oracle / opengrok

OpenGrok is a fast and usable source code search and cross reference engine, written in Java

http://oracle.github.io/opengrok/

Other

4.29k stars 739 forks source link

set the token size limit and align it with lucene #4470

Open tarzanek opened 8 months ago

tarzanek commented 8 months ago

since I cannot reopen https://github.com/oracle/opengrok/issues/4170

I am filing a new bug

above bug was not fixed, since you only upgraded jflex BUT you need to go to respective analyzers that used the workaround , or even to ALL analyzers and set the

%token_size_limit (<identifier>|<number>)

and align it with current lucenes limits (check their tokenizers, I think they also use jflex)

@ginoaugustine FYI