search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.49k stars 24.89k forks source link

[CI] UpgradeClusterClientYamlTestSuiteIT test {p0=mixed_cluster/10_basic/Start scroll in mixed cluster on upgraded node that we will continue after upgrade} failing #117206

Open elasticsearchmachine opened 1 week ago

elasticsearchmachine commented 1 week ago

Build Scans:

Reproduction Line:

./gradlew ":x-pack:qa:rolling-upgrade:v8.1.3#twoThirdsUpgradedTest" -Dtests.class="org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=mixed_cluster/100_analytics_usage/Basic test for usage stats on analytics indices}" -Dtests.seed=31D5B83898963B81 -Dtests.bwc=true -Dtests.locale=en-SD -Dtests.timezone=Australia/Currie -Druntime.java=23

Applicable branches: 8.16

Reproduces locally?: N/A

Failure History: See dashboard&_a=(controlGroupState:(initialChildControlState:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,fieldName:task.keyword,order:0,selectedOptions:!(),title:'GradleTask',type:optionsListControl),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,fieldName:className.keyword,order:1,selectedOptions:!(org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT),title:'Suite',type:optionsListControl),'144933da-5c1b-4257-a969-7f43455a7901':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,fieldName:name.keyword,order:2,selectedOptions:!(test%20%7Bp0%3Dmixed_cluster%2F10_basic%2FStart%20scroll%20in%20mixed%20cluster%20on%20upgraded%20node%20that%20we%20will%20continue%20after%20upgrade%7D),title:'Test',type:optionsListControl)))))

Failure Message:

java.lang.AssertionError: Failure at [mixed_cluster/100_analytics_usage:64]: field [analytics.stats.cumulative_cardinality_usage] is not greater than [$cumulative_cardinality_usage]
Expected: a value greater than <1>
     but: <1> was equal to <1>

Issue Reasons:

Note: This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

elasticsearchmachine commented 1 week ago

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

elasticsearchmachine commented 1 week ago

Pinging @elastic/es-search (Team:Search)

ywangd commented 1 week ago

I am rerouting this to the search team since the issue title says "Start scroll in mixed cluster ...". However the reproduction line says "100_analytics_usage/Basic test for usage stats on analytics indices". Based on the failure message, it is possible the that analytics usage test failure is the root cause since it has a meaningful assertion failure while the "start scroll" test failed with waiting for watcher nodes to become ready. I'll leave it to the search team for further triage.

elasticsearchmachine commented 1 week ago

Pinging @elastic/es-search-foundations (Team:Search Foundations)

benwtrent commented 1 week ago

So, the particular failure for search is due to a failure waiting for watcher to be ready. But, that is due to a node not being fully loaded due to an action tripping an assertion: https://github.com/elastic/elasticsearch/issues/117204

I am relabeling as ML as this issue likely has the same root cause as the linked issue.

elasticsearchmachine commented 1 week ago

Pinging @elastic/ml-core (Team:ML)

benwtrent commented 1 week ago

@davidkyle Looks like https://github.com/elastic/elasticsearch/pull/116236 needs its backport fixed and applied? We got a flurry of failed tests due to it on 8.16

davidkyle commented 1 week ago

Thanks @benwtrent I missed the failed backport

I opened https://github.com/elastic/elasticsearch/pull/117269