search

elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.42k stars 24.57k forks source link

[CI] FrozenSearchableSnapshotsIntegTests testCreateAndRestorePartialSearchableSnapshot failing #110408

Open DaveCTurner opened 2 months ago

DaveCTurner commented 2 months ago

Build scan: https://gradle-enterprise.elastic.co/s/aypliammakd7q/tests/:x-pack:plugin:searchable-snapshots:internalClusterTest/org.elasticsearch.xpack.searchablesnapshots.FrozenSearchableSnapshotsIntegTests/testCreateAndRestorePartialSearchableSnapshot

Reproduction line:

./gradlew ':x-pack:plugin:searchable-snapshots:internalClusterTest' --tests "org.elasticsearch.xpack.searchablesnapshots.FrozenSearchableSnapshotsIntegTests.testCreateAndRestorePartialSearchableSnapshot" -Dtests.seed=278167E3FC1AC094 -Dtests.locale=ar-QA -Dtests.timezone=Etc/GMT+3 -Druntime.java=22

Applicable branches: main

Reproduces locally?: Didn't try

Failure history: Failure dashboard for org.elasticsearch.xpack.searchablesnapshots.FrozenSearchableSnapshotsIntegTests#testCreateAndRestorePartialSearchableSnapshot&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('testCreateAndRestorePartialSearchableSnapshot'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.xpack.searchablesnapshots.FrozenSearchableSnapshotsIntegTests'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))

Failure excerpt:

java.lang.AssertionError: cannot complete future on thread Thread[#97,elasticsearch[node_s2][generic][T#2],5,TGRP-FrozenSearchableSnapshotsIntegTests] with waiter on thread Thread[#199,elasticsearch[node_s2][generic][T#8],5,TGRP-FrozenSearchableSnapshotsIntegTests], could deadlock if pool was full
    at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269)
    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:756)
    at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1126)
    at org.elasticsearch.action.support.PlainActionFuture$Sync.get(PlainActionFuture.java:262)
    at org.elasticsearch.action.support.PlainActionFuture.get(PlainActionFuture.java:74)
    at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:65)
    at org.elasticsearch.action.support.PlainActionFuture.actionGet(PlainActionFuture.java:171)
    at org.elasticsearch.xpack.searchablesnapshots.cache.blob.BlobStoreCacheService.get(BlobStoreCacheService.java:126)
    at org.elasticsearch.xpack.searchablesnapshots.store.SearchableSnapshotDirectory.getCachedBlob(SearchableSnapshotDirectory.java:673)
    at org.elasticsearch.xpack.searchablesnapshots.store.input.MetadataCachingIndexInput.readWithBlobCache(MetadataCachingIndexInput.java:280)
    at org.elasticsearch.xpack.searchablesnapshots.store.input.MetadataCachingIndexInput.readInternal(MetadataCachingIndexInput.java:615)
    at org.elasticsearch.blobcache.common.BlobCacheBufferedIndexInput.refill(BlobCacheBufferedIndexInput.java:304)
    at org.elasticsearch.blobcache.common.BlobCacheBufferedIndexInput.readByte(BlobCacheBufferedIndexInput.java:57)
    at org.apache.lucene.codecs.CodecUtil.readBEInt(CodecUtil.java:668)
    at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:185)
    at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:254)
    at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer.<init>(Lucene90DocValuesProducer.java:114)
    at org.apache.lucene.codecs.lucene90.Lucene90DocValuesFormat.fieldsProducer(Lucene90DocValuesFormat.java:154)
    at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.<init>(PerFieldDocValuesFormat.java:305)
    at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat.fieldsProducer(PerFieldDocValuesFormat.java:374)
    at org.apache.lucene.index.SegmentDocValues.newDocValuesProducer(SegmentDocValues.java:52)
    at org.apache.lucene.index.SegmentDocValues.getDocValuesProducer(SegmentDocValues.java:69)
    at org.apache.lucene.index.SegmentDocValuesProducer.<init>(SegmentDocValuesProducer.java:66)
    at org.apache.lucene.index.SegmentReader.initDocValuesProducer(SegmentReader.java:194)
    at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:113)
    at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:94)
    at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:77)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:820)
    at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)
    at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:67)
    at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:60)
    at org.elasticsearch.index.engine.ReadOnlyEngine.openDirectory(ReadOnlyEngine.java:575)
    at org.elasticsearch.index.engine.frozen.FrozenEngine.<init>(FrozenEngine.java:89)
    at org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshots.lambda$getEngineFactory$6(SearchableSnapshots.java:457)
    at org.elasticsearch.index.shard.IndexShard.createEngine(IndexShard.java:2150)
    at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:2123)
    at org.elasticsearch.index.shard.IndexShard.openEngineAndSkipTranslogRecovery(IndexShard.java:2099)
    at org.elasticsearch.indices.recovery.RecoveryTarget.lambda$prepareForTranslogOperations$1(RecoveryTarget.java:378)
    at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:348)
    at org.elasticsearch.indices.recovery.RecoveryTarget.prepareForTranslogOperations(RecoveryTarget.java:375)
    at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$4.handleRequest(PeerRecoveryTargetService.java:192)
    at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$4.handleRequest(PeerRecoveryTargetService.java:185)
    at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRequestHandler.messageReceived(PeerRecoveryTargetService.java:648)
    at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRequestHandler.messageReceived(PeerRecoveryTargetService.java:635)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
    at org.elasticsearch.transport.InboundHandler.doHandleRequest(InboundHandler.java:287)
    at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:300)
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1570)

  at __randomizedtesting.SeedInfo.seed([278167E3FC1AC094]:0)
  at org.elasticsearch.action.support.PlainActionFuture.assertCompleteAllowed(PlainActionFuture.java:416)
  at org.elasticsearch.action.support.PlainActionFuture.set(PlainActionFuture.java:137)
  at org.elasticsearch.action.support.PlainActionFuture.onResponse(PlainActionFuture.java:37)
  at org.elasticsearch.xpack.searchablesnapshots.cache.blob.BlobStoreCacheService$1.onFailure(BlobStoreCacheService.java:191)
  at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
  at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)
  at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31)
  at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39)
  at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
  at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)
  at org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:402)
  at org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:214)
  at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
  at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)
  at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31)
  at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onFailure(ActionListenerImplementations.java:317)
  at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
  at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)
  at org.elasticsearch.action.ActionListener$3.onFailure(ActionListener.java:402)
  at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:216)
  at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:195)
  at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$1.handleException(TransportSingleShardAction.java:240)
  at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1490)
  at org.elasticsearch.transport.TransportService$5.doRun(TransportService.java:1001)
  at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
  at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  at java.lang.Thread.run(Thread.java:1570)
elasticsearchmachine commented 2 months ago

Pinging @elastic/es-distributed (Team:Distributed)

idegtiarenko commented 1 month ago

Low risk as this is a fairly old test affected by a new stricter assertion