search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.72k stars 24.67k forks source link

AssertionError (lost uuid) in mixed cluster test #86889

Closed DaveCTurner closed 8 months ago

DaveCTurner commented 2 years ago

This test run failed due to a tripped assertion in one of the 7.0.1 nodes (#86887) but then one of the 7.17.4 nodes also fell over for apparently-unrelated reasons:

[2022-05-18T12:15:22,564][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [v7.0.1-0] fatal error in thread [elasticsearch[v7.0.1-0][snapshot][T#3]], exiting
java.lang.AssertionError: lost uuid JCGi3zjyQgytAyClMFs3Gg
        at org.elasticsearch.repositories.RepositoryData.snapshotsToXContent(RepositoryData.java:693) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.repositories.RepositoryData.snapshotsToXContent(RepositoryData.java:648) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$65(BlobStoreRepository.java:2323) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer.writeToPath(FsBlobContainer.java:274) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.common.blobstore.fs.FsBlobContainer.writeBlob(FsBlobContainer.java:253) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeAtomic(BlobStoreRepository.java:2636) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$66(BlobStoreRepository.java:2321) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:100) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:131) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:139) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:52) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:29) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$63(BlobStoreRepository.java:2300) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_301]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_301]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_301]
        Suppressed: java.lang.IllegalStateException: Failed to close the XContentBuilder
                at org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1131) ~[elasticsearch-x-content-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$65(BlobStoreRepository.java:2322) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.common.blobstore.fs.FsBlobContainer.writeToPath(FsBlobContainer.java:274) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.common.blobstore.fs.FsBlobContainer.writeBlob(FsBlobContainer.java:253) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.repositories.blobstore.BlobStoreRepository.writeAtomic(BlobStoreRepository.java:2636) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$66(BlobStoreRepository.java:2321) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:113) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:100) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:131) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:139) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:52) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:29) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$writeIndexGen$63(BlobStoreRepository.java:2300) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_301]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_301]
                at java.lang.Thread.run(Thread.java:748) [?:1.8.0_301]
        Caused by: java.io.IOException: Unclosed object or array found
                at org.elasticsearch.xcontent.json.JsonXContentGenerator.close(JsonXContentGenerator.java:456) ~[elasticsearch-x-content-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                at org.elasticsearch.xcontent.XContentBuilder.close(XContentBuilder.java:1129) ~[elasticsearch-x-content-7.17.4-SNAPSHOT.jar:7.17.4-SNAPSHOT]
                ... 19 more

Build scan: https://gradle-enterprise.elastic.co/s/hhxbspduxkiio/tests/:qa:mixed-cluster:v7.0.1%23mixedClusterTest/org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT/test%20%7Bp0=search.aggregation%2F230_composite%2FTerms%20source%20from%20sorted%7D

Reproduction line: ./gradlew ':qa:mixed-cluster:v7.0.1#mixedClusterTest' -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=search.aggregation/230_composite/Terms source from sorted}" -Dtests.seed=12FB47E08AFF8223 -Dtests.bwc=true -Dtests.locale=el-CY -Dtests.timezone=Atlantic/Stanley -Druntime.java=8

Applicable branches: 7.17

Reproduces locally?: Didn't try

Failure history: https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT&tests.test=test%20%7Bp0%3Dsearch.aggregation/230_composite/Terms%20source%20from%20sorted%7D

elasticmachine commented 2 years ago

Pinging @elastic/es-distributed (Team:Distributed)

original-brownbear commented 2 years ago

relates https://github.com/elastic/elasticsearch/issues/87656

mark-vieira commented 8 months ago

Two builds failed with this same error today so I'm recategorizing this as a blocker so we can retriage this.

https://gradle-enterprise.elastic.co/s/jpnxbhwsb6iac/console-log/task/:qa:mixed-cluster:v7.0.1%23mixedClusterTest?page=1 https://gradle-enterprise.elastic.co/s/ob7ocr3ilavps/console-log/task/:qa:mixed-cluster:v7.3.1%23mixedClusterTest?page=1

DaveCTurner commented 8 months ago

Looks like a genuine bug, although the impact should be fairly minor: I believe it could stop snapshots from working temporarily during a rolling upgrade from <7.5.0, but won't cause lasting damage and things should return to normal once the upgrade finishes.

relates https://github.com/elastic/elasticsearch/issues/87656

I don't think it's this.

DaveCTurner commented 8 months ago

Closed by https://github.com/elastic/elasticsearch/pull/104354