search

mikemccand / stargazers-migration-test

Testing Lucene's Jira -> GitHub issues migration
0 stars 0 forks source link

Use incoming thread for execution if IndexSearcher has an executor [LUCENE-8865] #862

Closed mikemccand closed 5 years ago

mikemccand commented 5 years ago

Today we don't utilize the incoming thread for a search when IndexSearcher has an executor. This thread is only idleing but can be used to execute a search once all other collectors are dispatched.

Legacy Jira details

LUCENE-8865 by Simon Willnauer (@s1monw) on Jun 17 2019, resolved Jun 18 2019

mikemccand commented 5 years ago

+1

[Legacy Jira: Uwe Schindler (@uschindler) on Jun 17 2019]

mikemccand commented 5 years ago

Commit 60f3b25d0600b1a1d3ea93f6f31d3bff02d041a9 in lucene-solr's branch refs/heads/master from Simon Willnauer https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=60f3b25

LUCENE-8865: Use incoming thread for execution if IndexSearcher has an executor (#725)

Today we don't utilize the incoming thread for a search when IndexSearcher has an executor. This thread is only idling but can be used to execute a search once all other collectors are dispatched.

[Legacy Jira: ASF subversion and git services on Jun 18 2019]

mikemccand commented 5 years ago

Commit 4165184b16027d5b66f3ad97806cf6e4425e466f in lucene-solr's branch refs/heads/branch_8x from Simon Willnauer https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=4165184

LUCENE-8865: Use incoming thread for execution if IndexSearcher has an executor (#725)

Today we don't utilize the incoming thread for a search when IndexSearcher has an executor. This thread is only idling but can be used to execute a search once all other collectors are dispatched.

[Legacy Jira: ASF subversion and git services on Jun 18 2019]

mikemccand commented 5 years ago

Commit c6899fc40d71fe6343f8d83c02641501aaaea049 in lucene-solr's branch refs/heads/master from Simon Willnauer https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c6899fc

LUCENE-8865: Move to executor in IndexSearcher (#731)

In order to simplify testing this change moves to use the Executor interface instead of ExecutorService. This change also simplifies customizing execute methods for use-cases that need to add additional logic for forking to new threads. This change also adds a test for the optimization added in LUCENE-8865.

This change is fully backwards compatible since ExecutorService implements Executor.

[Legacy Jira: ASF subversion and git services on Jun 20 2019]

mikemccand commented 5 years ago

Commit c6899fc40d71fe6343f8d83c02641501aaaea049 in lucene-solr's branch refs/heads/master from Simon Willnauer https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c6899fc

LUCENE-8865: Move to executor in IndexSearcher (#731)

In order to simplify testing this change moves to use the Executor interface instead of ExecutorService. This change also simplifies customizing execute methods for use-cases that need to add additional logic for forking to new threads. This change also adds a test for the optimization added in LUCENE-8865.

This change is fully backwards compatible since ExecutorService implements Executor.

[Legacy Jira: ASF subversion and git services on Jun 20 2019]

mikemccand commented 5 years ago

Commit 741207da14607969620923e2f77d64bd811fd350 in lucene-solr's branch refs/heads/branch_8x from Simon Willnauer https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=741207d

LUCENE-8865: Move to executor in IndexSearcher (#731)

In order to simplify testing this change moves to use the Executor interface instead of ExecutorService. This change also simplifies customizing execute methods for use-cases that need to add additional logic for forking to new threads. This change also adds a test for the optimization added in LUCENE-8865.

This change is fully backwards compatible since ExecutorService implements Executor.

[Legacy Jira: ASF subversion and git services on Jun 20 2019]

mikemccand commented 5 years ago

Commit 741207da14607969620923e2f77d64bd811fd350 in lucene-solr's branch refs/heads/branch_8x from Simon Willnauer https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=741207d

LUCENE-8865: Move to executor in IndexSearcher (#731)

In order to simplify testing this change moves to use the Executor interface instead of ExecutorService. This change also simplifies customizing execute methods for use-cases that need to add additional logic for forking to new threads. This change also adds a test for the optimization added in LUCENE-8865.

This change is fully backwards compatible since ExecutorService implements Executor.

[Legacy Jira: ASF subversion and git services on Jun 20 2019]

mikemccand commented 5 years ago

Using Executor is also much better if you want a completely different type of thread pool. Having the most abstract and simple interface for submitting/executing jobs is nice.

[Legacy Jira: Uwe Schindler (@uschindler) on Jun 20 2019]

mikemccand commented 5 years ago

I randomly came across the issue, this is a nice change! Were you able to measure the improvement?

[Legacy Jira: Tony Xu on Jun 24 2019]

mikemccand commented 5 years ago

@hypothesisx86 I didn't run any benchmarks. maybe @mikemccand can provide infos if there are improvements.

[Legacy Jira: Simon Willnauer (@s1monw) on Jun 25 2019]

mikemccand commented 5 years ago

I plan to test this using our production benchmarks ... will try to do that soon.

[Legacy Jira: Michael McCandless (@mikemccand) on Jun 26 2019]

mikemccand commented 5 years ago

Alas, I ran our internal benchmarks (production queries on production documents, measuring red-line QPS and long-pole query latencies at 10% capacity) and I could not measure any change due to this fix – it seems to be in the noise.

I was hoping for a small gain due to one fewer thread context switch ... but I still think the change is a good one!  Thanks @s1monw!

[Legacy Jira: Michael McCandless (@mikemccand) on Jul 22 2019]

mikemccand commented 5 years ago

Closing after the 8.2.0 release

[Legacy Jira: Ignacio Vera (@iverase) on Jul 26 2019]