search

google / zoekt

Fast trigram based code search
1.67k stars 113 forks source link

Need to exempt certain file types from being treated as binary due to trigram count #138

Open robinp-tw opened 2 years ago

robinp-tw commented 2 years ago

Files larger than approx the max_trigram_count (default 20k) are treated as binary, thus excluded from content-based indexing.

If you search for lang:binary, there's chance you'll run into some large source codes or other content you still would wish to index.

The logic lives in https://github.com/google/zoekt/blob/master/indexbuilder.go#L298 . Could maybe plumb some options that excludes files of certain pattern from the "too long, probably binary" treatment (raising the trigram limit is not really as option, there's always just one more file that is above any limit and would have been nice to index).

robinp-tw commented 2 years ago

Actually filesize-based exemption is already available, and the limit is 2MB by default. What I stumbled into is indeed the max_trigram_count. Maybe that could have a filename-based exemption similar to https://github.com/google/zoekt/blob/8d919baff972610a8eb4d57bf738d8f159bf8941/build/builder.go#L302.