search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.6k stars 24.63k forks source link

[CI] SearchStatesIT testBWCSearchStates failing #96436

Open slobodanadamovic opened 1 year ago

slobodanadamovic commented 1 year ago 
» [2023-05-30T13:12:01,041][WARN ][o.e.c.c.PublicationTransportHandler] [v8.8.0-local-1] unexpected error while deserializing an incoming cluster state java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.util.Map (java.lang.Integer and java.util.Map are in module java.base of loader 'bootstrap')
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:696)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.cluster.metadata.ComposableIndexTemplate.<init>(ComposableIndexTemplate.java:175)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:639)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:623)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.cluster.metadata.ComposableIndexTemplateMetadata.<init>(ComposableIndexTemplateMetadata.java:63)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:56)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.cluster.NamedDiffableValueSerializer.read(NamedDiffableValueSerializer.java:29)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.cluster.NamedDiffableValueSerializer.read(NamedDiffableValueSerializer.java:19)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.cluster.DiffableUtils$MapDiff.<init>(DiffableUtils.java:317)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.cluster.DiffableUtils.readImmutableOpenMapDiff(DiffableUtils.java:94)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.cluster.metadata.Metadata$MetadataDiff.<init>(Metadata.java:1548)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.cluster.metadata.Metadata.readDiffFrom(Metadata.java:1408)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.cluster.ClusterState$ClusterStateDiff.<init>(ClusterState.java:1023)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.cluster.ClusterState.readDiffFrom(ClusterState.java:900)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.cluster.coordination.PublicationTransportHandler.handleIncomingPublishRequest(PublicationTransportHandler.java:164)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$new$0(PublicationTransportHandler.java:109)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:74)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:315)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
»   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
»   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
»   at java.base/java.lang.Thread.run(Thread.java:1623)
»

Build scan: https://gradle-enterprise.elastic.co/s/elfqagawbnyws/tests/:qa:ccs-rolling-upgrade-remote-cluster:v8.8.0%23oldClusterTest/org.elasticsearch.upgrades.SearchStatesIT/testBWCSearchStates

Reproduction line:

./gradlew ':qa:ccs-rolling-upgrade-remote-cluster:v8.8.0#oldClusterTest' -Dtests.class="org.elasticsearch.upgrades.SearchStatesIT" -Dtests.method="testBWCSearchStates" -Dtests.seed=315F640F0309F61B -Dtests.bwc=true -Dtests.locale=sv-SE -Dtests.timezone=Australia/Melbourne -Druntime.java=20

Applicable branches: main

Reproduces locally?: Didn't try

Failure history: https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.upgrades.SearchStatesIT&tests.test=testBWCSearchStates

Failure excerpt:

java.net.ConnectException: Connection refused

  at __randomizedtesting.SeedInfo.seed([315F640F0309F61B:C4C9C4E2180C9574]:0)
  at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:930)
  at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
  at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
  at org.elasticsearch.test.rest.ESRestTestCase.createIndex(ESRestTestCase.java:1576)
  at org.elasticsearch.test.rest.ESRestTestCase.createIndex(ESRestTestCase.java:1537)
  at org.elasticsearch.upgrades.SearchStatesIT.testBWCSearchStates(SearchStatesIT.java:189)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  at java.lang.reflect.Method.invoke(Method.java:578)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1623)

  Caused by: java.net.ConnectException: Connection refused

    at sun.nio.ch.Net.pollConnect(Net.java:-2)
    at sun.nio.ch.Net.pollConnectNow(Net.java:673)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973)
    at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
    at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
    at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
    at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
    at java.lang.Thread.run(Thread.java:1623)
elasticsearchmachine commented 1 year ago

Pinging @elastic/es-search (Team:Search)

edsavage commented 1 year ago

Another one: https://gradle-enterprise.elastic.co/s/73caiutbvz2rw

ywangd commented 1 year ago

Happened again https://gradle-enterprise.elastic.co/s/aywdihzt4wrm4

dnhatn commented 1 year ago 
»  Caused by: java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.util.Map (java.lang.Integer and java.util.Map are in module java.base of loader 'bootstrap')
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:696)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.cluster.metadata.ComposableIndexTemplate.<init>(ComposableIndexTemplate.java:175)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.action.admin.indices.template.put.PutComposableIndexTemplateAction$Request.<init>(PutComposableIndexTemplateAction.java:54)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.transport.RequestHandlerRegistry.newRequest(RequestHandlerRegistry.java:61)
»   at org.elasticsearch.server@8.8.0/org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:273)
benwtrent commented 11 months ago

Actually, this hasn't failed for quite a long time (all failures since Aug, were in PRs making CCS changes, so it makes sense for a CCS test to cause issues :)).

Next time it fails we should export the logs and verify the cause. If it is indeed composable templates, that indicates it has nothing to do with this particular test, but instead is a BWC issue with templates.

Until then, I am going to close.

albertzaharovits commented 6 months ago

Here are a couple of recent ones: https://gradle-enterprise.elastic.co/s/af2qeczdb7kqc https://gradle-enterprise.elastic.co/s/6s2sgf6hvcu2e https://gradle-enterprise.elastic.co/s/oleqsgspwjz7q

It seems to fail consistently.

build stats: https://es-delivery-stats.elastic.dev/app/dashboards#/view/dcec9e60-72ac-11ee-8f39-55975ded9e63?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:'2024-02-29T22:00:00.000Z',to:now))

albertzaharovits commented 6 months ago

Going to try and find some logs.

albertzaharovits commented 6 months ago

curl -L -H 'Authorization: f5610ca828b39596' \ -o '2170(1).tar.bz2' \ https://upload.elastic.co/d/54293a18baa76c91c62cdbf148d925250d50fe4bccd90a9268b8dc74dcc61943

curl -L -H 'Authorization: e8c13084b8b2d59a' \ -o '2170(2).tar.bz2' \ https://upload.elastic.co/d/5d28b97d5916ba7edb87f36b3bdcae2340c2a42afb1a2cfceba35a720e7263a1

curl -L -H 'Authorization: 3c6f8d989dce6b40' \ -o '2170.tar.bz2' \ https://upload.elastic.co/d/a68c9e81a5daff90f4a4d623b3123539e6e8e78fd405ca74f31ba798891f505b

Valid for 360 days

piergm commented 5 months ago

Single test failing sporadically, assigned to myself and changed label to medium-risk

elasticsearchmachine commented 2 months ago

Pinging @elastic/es-search-foundations (Team:Search Foundations)