Open Luegg opened 2 years ago
@Luegg thanks. Have you tried running it without track_total_hits
being sent to get a sense of how much faster it is? Also for what it's worth, we're working on an optimization that delegates all of this logic to Elasticsearch so that it's just a single query rather than fetching all data -- #1801.
Yes, it took around 10min to fetch all the docs. Which boils down to ~60 requests/s.
Issue description
While using SparkSQL with the ES connector I noticed that all the search requests that hit ES use
"track_total_hits": 2147483647
. This seems to have a big impact on query performance, especially for queries that scroll through large datasets.Steps to reproduce
The following reproduction uses the NOAA dataset from https://github.com/elastic/rally-tracks/tree/master/sql to demonstrate the impact on performance:
Setup
Spark Shell
The ES slowlog shows increasing
elasticsearch.slowlog.took
times for the search queries:Altogether, the query fetching 33659481 docs did not finish within 3 hours eventually reaching
took
times of > 2s.Version Info
OS: : OSX JVM : 11.0.14-tem Hadoop/Spark: Spark 3.2.1 ES-Hadoop : elasticsearch-spark-30_2.12-8.1.0.jar ES : 8.1.0