[CI] MixedClusterClientYamlTestSuiteIT test {p0=search.aggregation/240_max_buckets/Max bucket} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #4021 / 7.5.2_bwc
• elasticsearch-periodic #3988 / 7.3.2_bwc

Reproduction Line:

./gradlew ':qa:mixed-cluster:v7.5.2#mixedClusterTest' -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=search.aggregation/240_max_buckets/Max bucket}" -Dtests.seed=64F14D074066F266 -Dtests.bwc=true -Dtests.locale=es-NI -Dtests.timezone=America/Creston -Druntime.java=22

Applicable branches: 7.17

Reproduces locally?: N/A

Failure History: See dashboard&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('test%20%7Bp0%3Dsearch.aggregation%2F240_max_buckets%2FMax%20bucket%7D'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))

Failure Message:

java.lang.RuntimeException: Failure at [search.aggregation/240_max_buckets:70]: Connection refused

Issue Reasons:

• [7.17] 2 failures in test test {p0=search.aggregation/240_max_buckets/Max bucket} (1.3% fail rate in 152 executions)
• [7.17] 2 failures in pipeline elasticsearch-periodic (16.7% fail rate in 12 executions)

Note: This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[astefan]

I am not sure why Connection refused is the error message in the description. Looking at the console log, I see that one but also this:

  1> [2024-09-11T05:33:47,847][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] [p0=search.aggregation/240_max_buckets/Max bucket] before test
  1> [2024-09-11T05:33:48,040][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  1>   "stash" : {
  1>     "body" : {
  1>       "error" : {
  1>         "root_cause" : [ ],
  1>         "type" : "search_phase_execution_exception",
  1>         "reason" : "",
  1>         "phase" : "fetch",
  1>         "grouped" : true,
  1>         "failed_shards" : [ ],
  1>         "caused_by" : {
  1>           "type" : "too_many_buckets_exception",
  1>           "reason" : "Trying to create too many buckets. Must be less than or equal to: [3] but this number of buckets was exceeded. This limit can be set by changing the [search.max_buckets] cluster level setting.",
  1>           "max_buckets" : 3,
  1>           "stack_trace" : "TooManyBucketsException[Trying to create too many buckets. Must be less than or equal to: [3] but this number of buckets was exceeded. This limit can be set by changing the [search.max_buckets] cluster level setting.]
  1>    at org.elasticsearch.search.aggregations.MultiBucketConsumerService$MultiBucketConsumer.accept(MultiBucketConsumerService.java:115)
  1>    at org.elasticsearch.search.aggregations.InternalAggregation$ReduceContext.consumeBucketsAndMaybeBreak(InternalAggregation.java:163)
  1>    at org.elasticsearch.search.aggregations.DelayedBucket.reduced(DelayedBucket.java:58)
  1>    at org.elasticsearch.search.aggregations.TopBucketBuilder$PriorityQueueTopBucketBuilder.build(TopBucketBuilder.java:128)
  1>    at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduce(AbstractInternalTerms.java:313)
  1>    at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:253)
  1>    at org.elasticsearch.search.aggregations.InternalAggregations.topLevelReduce(InternalAggregations.java:188)
  1>    at org.elasticsearch.action.search.SearchPhaseController.reduceAggs(SearchPhaseController.java:538)
  1>    at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:507)
  1>    at org.elasticsearch.action.search.QueryPhaseResultConsumer.reduce(QueryPhaseResultConsumer.java:132)
  1>    at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:112)
  1>    at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:37)
  1>    at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:98)
  1>    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
  1>    at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
  1>    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777)
  1>    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
  1>    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  1>    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  1>    at java.base/java.lang.Thread.run(Thread.java:833)
  1> "
  1>         },
  1>         "stack_trace" : "Failed to execute phase [fetch], 
  1>    at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:713)
  1>    at org.elasticsearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:103)
  1>    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)
  1>    at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
  1>    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777)
  1>    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
  1>    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  1>    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  1>    at java.base/java.lang.Thread.run(Thread.java:833)
  1> Caused by: TooManyBucketsException[Trying to create too many buckets. Must be less than or equal to: [3] but this number of buckets was exceeded. This limit can be set by changing the [search.max_buckets] cluster level setting.]
  1>    at org.elasticsearch.search.aggregations.MultiBucketConsumerService$MultiBucketConsumer.accept(MultiBucketConsumerService.java:115)
  1>    at org.elasticsearch.search.aggregations.InternalAggregation$ReduceContext.consumeBucketsAndMaybeBreak(InternalAggregation.java:163)
  1>    at org.elasticsearch.search.aggregations.DelayedBucket.reduced(DelayedBucket.java:58)
  1>    at org.elasticsearch.search.aggregations.TopBucketBuilder$PriorityQueueTopBucketBuilder.build(TopBucketBuilder.java:128)
  1>    at org.elasticsearch.search.aggregations.bucket.terms.AbstractInternalTerms.reduce(AbstractInternalTerms.java:313)
  1>    at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:253)
  1>    at org.elasticsearch.search.aggregations.InternalAggregations.topLevelReduce(InternalAggregations.java:188)
  1>    at org.elasticsearch.action.search.SearchPhaseController.reduceAggs(SearchPhaseController.java:538)
  1>    at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:507)
  1>    at org.elasticsearch.action.search.QueryPhaseResultConsumer.reduce(QueryPhaseResultConsumer.java:132)
  1>    at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:112)
  1>    at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:37)
  1>    at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:98)
  1>    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
  1>    ... 6 more
  1> "
  1>       },
  1>       "status" : 503
  1>     }
  1>   }
  1> }]

[nik9000]

I am not sure why Connection refused is the error message in the description.

That's usually "the node died". We should figure out what's up with that.

[iverase]

down in the logs this is strange:

 [2024-09-11T12:30:31,243][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [v7.5.2-2] fatal error in thread [elasticsearch[v7.5.2-2][clusterApplierService#updateTask][T#1]], exiting
»  java.lang.AssertionError: {_doc=org.elasticsearch.index.mapper.DocumentMapper@379be8b1}
»   at org.elasticsearch.index.mapper.MapperService.assertMappingVersion(MapperService.java:272) ~[elasticsearch-7.5.2.jar:7.5.2]
»   at org.elasticsearch.index.mapper.MapperService.updateMapping(MapperService.java:226) ~[elasticsearch-7.5.2.jar:7.5.2]

[not-napoleon]

Is this only failing on 7.x? I'm not sure medium-risk is appropriate if that's the case.

[nik9000]

Is this only failing on 7.x? I'm not sure medium-risk is appropriate if that's the case.

I believe this is indeed only failing on 7.17 and it's related to something odd with mappings. I think at this point it's not worth digging into.