search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.5k stars 24.89k forks source link

[CI] RandomExceptionCircuitBreakerIT testBreakerWithRandomExceptions failing #115867

Open Tim-Brooks opened 1 month ago

Tim-Brooks commented 1 month ago

CI Link

https://gradle-enterprise.elastic.co/s/c6s5xwlb4qzhe

Repro line

./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.indices.memory.breaker.RandomExceptionCircuitBreakerIT" -Dtests.seed=A9617E1EDF4FAD00 -Dtests.locale=en -Dtests.timezone=Etc/UTC -Druntime.java=23

Does it reproduce?

Didn't try

Applicable branches

8.x

Failure history

No response

Failure excerpt

on.operations":"756","index.unassigned.node_left.delayed_timeout":"0"}]
  1> [2024-10-29T13:25:26,482][WARN ][o.e.d.c.s.Settings       ] [node_s2] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.bloom_filter_for_id_field.enabled" message="[index.bloom_filter_for_id_field.enabled] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2024-10-29T13:25:26,501][INFO ][o.e.c.m.MetadataCreateIndexService] [node_s2] [test] creating index, cause [api], templates [random_index_template], shards [7]/[1]
  1> [2024-10-29T13:25:26,538][WARN ][o.e.d.c.s.Settings       ] [node_s5] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.bloom_filter_for_id_field.enabled" message="[index.bloom_filter_for_id_field.enabled] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2024-10-29T13:25:26,546][WARN ][o.e.d.c.s.Settings       ] [node_s3] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.bloom_filter_for_id_field.enabled" message="[index.bloom_filter_for_id_field.enabled] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2024-10-29T13:25:26,555][WARN ][o.e.d.c.s.Settings       ] [node_s4] data_stream.dataset="deprecation.elasticsearch" data_stream.namespace="default" data_stream.type="logs" elasticsearch.event.category="settings" event.code="index.bloom_filter_for_id_field.enabled" message="[index.bloom_filter_for_id_field.enabled] setting was deprecated in Elasticsearch and will be removed in a future release."
  1> [2024-10-29T13:25:27,153][INFO ][o.e.c.r.a.AllocationService] [node_s2] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test][4]]])." previous.health="YELLOW" reason="shards started [[test][4]]"
  1> [2024-10-29T13:25:27,289][INFO ][o.e.i.m.b.RandomExceptionCircuitBreakerIT] [testBreakerWithRandomExceptions] Start Refresh
  1> [2024-10-29T13:25:27,492][INFO ][o.e.i.m.b.RandomExceptionCircuitBreakerIT] [testBreakerWithRandomExceptions] Refresh failed: [false] numShardsFailed: [0], shardFailuresLength: [0], successfulShards: [14], totalShards: [14] 
  1> [2024-10-29T13:25:27,694][INFO ][o.e.i.m.b.RandomExceptionCircuitBreakerIT] [testBreakerWithRandomExceptions] expected SearchPhaseException: [all shards failed]
  1> [2024-10-29T13:25:27,986][INFO ][o.e.i.m.b.RandomExceptionCircuitBreakerIT] [testBreakerWithRandomExceptions] expected SearchPhaseException: [all shards failed]
  1> [2024-10-29T13:25:28,199][INFO ][o.e.i.m.b.RandomExceptionCircuitBreakerIT] [testBreakerWithRandomExceptions] expected SearchPhaseException: [all shards failed]
  1> [2024-10-29T13:25:28,237][INFO ][o.e.i.m.b.RandomExceptionCircuitBreakerIT] [testBreakerWithRandomExceptions] expected SearchPhaseException: [all shards failed]
  1> [2024-10-29T13:25:28,352][INFO ][o.e.i.m.b.RandomExceptionCircuitBreakerIT] [testBreakerWithRandomExceptions] expected SearchPhaseException: [all shards failed]
  1> [2024-10-29T13:25:28,471][INFO ][o.e.i.m.b.RandomExceptionCircuitBreakerIT] [testBreakerWithRandomExceptions] expected SearchPhaseException: [all shards failed]
  1> [2024-10-29T13:25:28,559][INFO ][o.e.i.m.b.RandomExceptionCircuitBreakerIT] [testBreakerWithRandomExceptions] expected SearchPhaseException: [all shards failed]
  1> [2024-10-29T13:25:28,586][INFO ][o.e.i.m.b.RandomExceptionCircuitBreakerIT] [testBreakerWithRandomExceptions] expected SearchPhaseException: [all shards failed]
  1> [2024-10-29T13:25:28,663][INFO ][o.e.i.m.b.RandomExceptionCircuitBreakerIT] [testBreakerWithRandomExceptions] expected SearchPhaseException: [all shards failed]
  1> [2024-10-29T13:25:28,754][INFO ][o.e.i.m.b.RandomExceptionCircuitBreakerIT] [testBreakerWithRandomExceptions] expected SearchPhaseException: [all shards failed]
  1> [2024-10-29T13:25:28,792][INFO ][o.e.i.m.b.RandomExceptionCircuitBreakerIT] [testBreakerWithRandomExceptions] expected SearchPhaseException: [all shards failed]
  1> [2024-10-29T13:25:34,140][ERROR][o.e.t.LeakTracker        ] [[Cleaner-0]] LEAK: resource was not cleaned up before it was garbage-collected.
  1> Recent access records: 
  1> Created at:
  1>    in [TEST-RandomExceptionCircuitBreakerIT.testBreakerWithRandomExceptions-seed#[A9617E1EDF4FAD00]][testBreakerWithRandomExceptions]
  1>    org.elasticsearch.action.search.ArraySearchPhaseResults.<init>(ArraySearchPhaseResults.java:28)
  1>    org.elasticsearch.action.search.QueryPhaseResultConsumer.<init>(QueryPhaseResultConsumer.java:86)
  1>    org.elasticsearch.action.search.SearchPhaseController.newSearchPhaseResults(SearchPhaseController.java:841)
  1>    org.elasticsearch.action.search.TransportSearchAction$AsyncSearchActionProvider.newSearchPhase(TransportSearchAction.java:1500)
  1>    org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1305)
  1>    org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:1038)
  1>    org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$4(TransportSearchAction.java:365)
  1>    org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:247)
  1>    org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:109)
  1>    org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77)
  1>    org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:531)
  1>    org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:316)
  1>    org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:121)
  1>    org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:135)
  1>    org.elasticsearch.action.support.TransportAction.handleExecution(TransportAction.java:96)
  1>    org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:59)
  1>    org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:197)
  1>    org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:106)
  1>    org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:84)
  1>    org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:140)
  1>    org.elasticsearch.client.internal.FilterClient.doExecute(FilterClient.java:56)
  1>    org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:140)
  1>    org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:126)
elasticsearchmachine commented 1 month ago

Pinging @elastic/es-search (Team:Search)

elasticsearchmachine commented 1 month ago

Pinging @elastic/es-search-foundations (Team:Search Foundations)