elasticsearchmachine
commented
1 year ago Pinging @elastic/es-search (Team:Search)
Open iverase opened 1 year ago
elasticsearchmachine
commented
1 year ago Pinging @elastic/es-search (Team:Search)
droberts195
commented
1 year ago Failed again in https://gradle-enterprise.elastic.co/s/bfoh4lwoag6tq
benwtrent
commented
1 year ago This failure is weird. It seems to only be with the BWC with 8.6.0. The previous clause passes, and uses min_doc_count: 0. So, it makes sense no shards are skipped.
But then this failing clause occurs, and min_doc_count isn't provided. So, it should skip a shard.
I am digging to see if anything changed with terms and bwc with main and 8.6. Something weird is indeed happening that is indicating that shards cannot be skipped. Since this only happens periodically, I am thinking it has to do with which node is handling the request and which nodes are upgraded...
benwtrent
commented
1 year ago The stashed response when the test failed indicates that min_doc_count is indeed the default value of 1 as we don't receive any buckets with a 0 doc count.
"stash" : {
"body" : {
"took" : 16,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"idx_terms" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "index_2",
"doc_count" : 1
},
{
"key" : "index_3",
"doc_count" : 1
}
]
}
}
}
}
pxsalehi
commented
1 year ago There has been more failure of this, e.g.: https://gradle-enterprise.elastic.co/s/imkrolbzltpfq/tests/:qa:mixed-cluster:v8.6.2%23mixedClusterTest/org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT/test%20%7Bp0=search%2F140_pre_filter_search_shards%2Fpre_filter_shard_size%20with%20shards%20that%20have%20no%20hit%7D?top-execution=1
benwtrent
commented
1 year ago OK, the cause is still evading me.
This is earlier in the same test and it passes.
# this is a case where we can actually skip due to rewrite
- do:
search:
rest_total_hits_as_int: true
pre_filter_shard_size: 1
body: { "size" : 0, "query" : { "range" : { "created_at" : { "gte" : "2016-02-01", "lt": "2018-02-01"} } } }
- match: { _shards.total: 3 }
- match: { _shards.successful: 3 }
- match: { _shards.skipped : 1}
- match: { _shards.failed: 0 }
- match: { hits.total: 2 }And here is the failing query
- do:
search:
rest_total_hits_as_int: true
pre_filter_shard_size: 1
body: { "size" : 0, "query" : { "range" : { "created_at" : { "gte" : "2016-02-01", "lt": "2018-02-01"}}}, "aggs" : { "idx_terms" : { "terms" : { "field" : "_index" } } } }
- match: { _shards.total: 3 }
- match: { _shards.successful: 3 }
- match: { _shards.skipped : 1 } # Actual response has `0` here.
- match: { _shards.failed: 0 }
- match: { hits.total: 2 }
- length: { aggregations.idx_terms.buckets: 2 }The response doesn't have 0 buckets, so we know the min_doc_count: 1
So, it definitely has to do with the inclusion of the terms agg, or we are caching something that we shouldn't be.
Still investigating.
astefan
commented
1 year ago Another one here. Tried locally and it doesn't repro.
REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v8.6.3#mixedClusterTest' -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=search/140_pre_filter_search_shards/pre_filter_shard_size with shards that have no hit}" -Dtests.seed=9B95319E6806A1D6 -Dtests.bwc=true -Dtests.locale=zh-Hant-HK -Dtests.timezone=SystemV/EST5EDT -Druntime.java=18
org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT > test {p0=search/140_pre_filter_search_shards/pre_filter_shard_size with shards that have no hit} FAILED
java.lang.AssertionError: Failure at [search/140_pre_filter_search_shards:164]:
Expected: <1>
but: was <0>
at __randomizedtesting.SeedInfo.seed([9B95319E6806A1D6:13C10E44C6FACC2E]:0)
at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:547)
at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:499)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:577)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
cbuescher
commented
1 year ago 
iverase
commented
1 year ago Another one: https://gradle-enterprise.elastic.co/s/3ycgwkaofc6d4
valeriy42
commented
1 year ago Another failure here: https://gradle-enterprise.elastic.co/s/swoditvjvkl32
cbuescher
commented
1 year ago And today on main: https://gradle-enterprise.elastic.co/s/drgmcwegmz2l4
ywangd
commented
1 year ago droberts195
commented
1 year ago Another occurrence in https://gradle-enterprise.elastic.co/s/2vxxczqbrzggc
nik9000
commented
12 months ago 
ldematte
commented
11 months ago Happened again today, on 8.11 branch testing bwc to 8.10.4: https://gradle-enterprise.elastic.co/s/pnblflald6v6a/tests/task/:qa:mixed-cluster:v8.10.4%23mixedClusterTest/details/org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT/test%20%7Bp0=search%2F140_pre_filter_search_shards%2Fpre_filter_shard_size%20with%20shards%20that%20have%20no%20hit%7D?top-execution=1
Mute https://github.com/elastic/elasticsearch/pull/100954 should be backported to 8.11 (and 8.10 too probably)
ldematte
commented
11 months ago Failed also on 8.10, testing bwc to 8.6.2: https://gradle-enterprise.elastic.co/s/zi7pdhzsmxrks/tests/:qa:mixed-cluster:v8.6.2%23mixedClusterTest/org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT/test%20%7Bp0=search%2F140_pre_filter_search_shards%2Fpre_filter_shard_size%20with%20shards%20that%20have%20no%20hit%7D
Mute https://github.com/elastic/elasticsearch/pull/100954 should be backported to 8.11 and 8.10
elasticsearchmachine
commented
2 months ago Pinging @elastic/es-search-foundations (Team:Search Foundations)
Only run it once and did not reproduce.
Build scan: https://gradle-enterprise.elastic.co/s/xuwjw32yoth5u/tests/:qa:mixed-cluster:v8.6.0%23mixedClusterTest/org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT/test%20%7Bp0=search%2F140_pre_filter_search_shards%2Fpre_filter_shard_size%20with%20shards%20that%20have%20no%20hit%7D
Reproduction line:
Applicable branches: main
Reproduces locally?: No
Failure history: https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT&tests.test=test%20%7Bp0%3Dsearch/140_pre_filter_search_shards/pre_filter_shard_size%20with%20shards%20that%20have%20no%20hit%7D
Failure excerpt: