search

pat / thinking-sphinx

Sphinx/Manticore plugin for ActiveRecord/Rails
http://freelancing-gods.com/thinking-sphinx
MIT License
1.63k stars 468 forks source link

Searchd memory usage #1188

Closed atomical closed 3 years ago

atomical commented 3 years ago

Hi Pat,

Is there any way to limit memory usage? Currently using 6 gigs with a big index. A cursory search reveals it's not possible?

pat commented 3 years ago

Hi Adam - I don't think this is possible, as the Sphinx daemon loads all attribute data into memory. Hence: the more records (and more attributes) you have, the larger the memory.

I guess there's the option of splitting up the data into separate daemons/machines, but that's going to get complex quickly - using a single machine with more memory is almost certainly going to be the wiser path.

atomical commented 3 years ago

Yes, we picked single machine as our approach and that was a good idea. Ram is cheap -- except if it's cloud Ram. 😄

I'm actually wondering if running out of ram is the reason the searchd instance is crashing on every deploy. Then we have to remove the binlog and start the process again. We can do this without rebuilding so there isn't much downtime. But still it's not great.

[Tue Dec 22 12:18:48.912 2020] [14080] WARNING: binlog: update: unexpected tid (index=abc_core, indextid=0, logtid=7, pos=81668)
[Tue Dec 22 12:18:48.912 2020] [14080] WARNING: binlog: update: unexpected tid (index=abc2_core, indextid=0, logtid=20, pos=81935)
[Tue Dec 22 12:18:48.913 2020] [14080] WARNING: binlog: update: unexpected tid (index=abc2_core, indextid=0, logtid=116, pos=127173)
[Tue Dec 22 12:18:48.914 2020] [14080] WARNING: binlog: update: unexpected tid (index=abc3_module_core, indextid=0, logtid=67, pos=164857)
[Tue Dec 22 12:18:48.918 2020] [14080] WARNING: binlog: update: unexpected tid (index=abc4_core, indextid=0, logtid=39, pos=329940)
[Tue Dec 22 12:18:48.918 2020] [14080] FATAL: binlog: update: descending tid (index=abc5_core, lasttid=116, logtid=1, pos=330114)

We're using Sphinx 2.3.2-id64-beta.

pat commented 3 years ago

At a guess I don't think that's a RAM issue… I wonder if the daemon's being forcibly killed rather than shutting itself down properly (and thus the binlog files aren't quite right). Is there anything in the daemon log that suggests a rushed shutdown?

atomical commented 3 years ago

That's probably likely. Although I think it could be forcibly killed because of a ram issue? I haven't seen anything in the log but I'll see what I can dig up from the system.

pat commented 3 years ago

Yeah, I've seen daemons killed before if the machine's RAM is being maxed out. Not quite sure why I didn't think of that in my last message - my previous assertion about it not being a RAM issue is quite possibly wrong! 🤦🏻‍♂️