search

pat / thinking-sphinx

Sphinx/Manticore plugin for ActiveRecord/Rails
http://freelancing-gods.com/thinking-sphinx
MIT License
1.63k stars 468 forks source link

Make MAXIMUM_STATEMENT_LENGTH customizable #1178

Closed kalsan closed 3 years ago

kalsan commented 4 years ago

Hi Pat :-)

Sphinx 0.9.9 introduced the setting max_packet_size which defaults to 8M, limiting the length of a query that can be run in Sphinx ( http://sphinxsearch.com/docs/current/conf-max-packet-size.html ). ThinkingSphinx has a hardcoded value of MAXIMUM_STATEMENT_LENGTH = (2 ** 23) - 5 matching Sphinx' default limit. However, the value is hardcoded in lib/thinking_sphinx.rb:20 and cannot be changed.

Because I have huge amounts of text in a single document (over 8 MB of text), reducing the batch size to 1 would not help, thus I experimented by adding a custom max_packet_size to config/thinking_sphinx.yml and overwriting MAXIMUM_STATEMENT_LENGTH in a monkey patch. Result: Actions like callbacks, ts:rebuild, ts:index, which had crashed due to excessive QL length before, appear to work fine now.

For this reason, I think that MAXIMUM_STATEMENT_LENGTH should be customizable in Thinking Sphinx. Would you agree or are the problems I oversee?

Best, Kalsan

pat commented 4 years ago

Hi Kalsan,

I hadn't realised Sphinx allows this value to be customisable - but yes, given that's possible, it makes sense for Thinking Sphinx to allow the maximum to be customised as well. If you'd like to submit a pull request, that'd be great! :)

Thanks!

kalsan commented 4 years ago

Hi Pat

I did my best to create the pull request, hope it works for you.

Best Kalsan

pat commented 3 years ago

Closing this because the PR got merged in :D