searchd is single threaded

[GoogleCodeExporter]

What steps will reproduce the problem?
1. default install with install.sh on ubuntu server 10.04LTS
2. point firewall syslog data toward elsa
3. search for host=firewallipaddress

What is the expected output? What do you see instead?
Show near 30million results.  Shows 30million results but takes 15 seconds to 
return data.  Running top on the physical host, an 8 proc machine with 16GB of 
memory shows only one processor pegging at 100% running process searchd.

What version of the product are you using? On what operating system?
latest deploy from SVN - deployed April 27th 2012. Ubuntu 10.04 LTS

Please provide any additional information below.
I thought searchd was multithreaded?

Original issue reported on code.google.com by thecisco...@gmail.com on 2 May 2012 at 3:02

[GoogleCodeExporter]

Searchd is multithreaded, but only insofar as multiple searches can be run at 
the same time, each in their own thread.  An early version of ELSA took 
advantage of this by using distributed searches that looped back around to hit 
the same host (essentially load-balancing a given search across sub-searches 
within the same searchd instance using Sphinx distributed indexes).  However, 
maintaining what distributed indexes pointed at which real indexes as the 
contents of the real indexes shifted became problematic, and so it was 
abandoned in recent ELSA versions.  It's worth taking another look at, and I 
will put this down as a feature request and try to revisit this issue soon, as 
you raise a good point about not taking full advantage of multi-core 
architectures.

Original comment by mchol...@gmail.com on 2 May 2012 at 4:20

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Thanks for the detailed explanation.

Original comment by thecisco...@gmail.com on 2 May 2012 at 7:00

[GoogleCodeExporter]

This is now fixed as of r264.  Queries over 1 billion rows will auto-thread.  
This amount can be set with the config entry of "distributed_threshold."

Original comment by mchol...@gmail.com on 15 May 2012 at 6:22

• Changed state: Fixed

[GoogleCodeExporter]

Awesome,

Is there some documentation on how to upgrade?

Original comment by thecisco...@gmail.com on 16 May 2012 at 7:09

[GoogleCodeExporter]

Yep, just added a wiki page:
http://code.google.com/p/enterprise-log-search-and-archive/wiki/Updating

On Wed, May 16, 2012 at 2:10 PM,
<enterprise-log-search-and-archive@googlecode.com> wrote:

Original comment by mchol...@gmail.com on 16 May 2012 at 9:47

[GoogleCodeExporter]

Thanks for the documentation.  I'm having just one problem.

I get this error.

Exported revision 270.
get_elsa success
Executing update_web_mysql
Updating web MySQL...
ERROR 1091 (42000) at line 1: Can't DROP 'action_params'; check that column/key 
exists
update_web_mysql success
!!!!!! IMPORTANT !!!!!!!!!
If you have a host-based firewall like IPTables running, remember to allow 
ports 80 (and/or 443) for the web server and ports 514 (syslog), 3306 (MySQL), 
and 3307 (Sphinx) for log nodes

Any ideas?   I have set the root mysql password at the top of the install script

Original comment by thecisco...@gmail.com on 17 May 2012 at 8:38

[GoogleCodeExporter]

Ah, this isn't clear without checking the install.sh source, but you can ignore 
the MySQL errors during the update as not all commands will apply to all 
versions.  When I write the update commands, I make sure they are backwards 
compatible such that the failure won't break anything.  I've updated install.sh 
to have a message indicating this.

Original comment by mchol...@gmail.com on 17 May 2012 at 8:45

[GoogleCodeExporter]

Cool, everything appears to be working correctly.

Thanks again.

Cheers.

Original comment by thecisco...@gmail.com on 17 May 2012 at 8:47