search

FoundationDB / fdb-record-layer

A record-oriented store built on FoundationDB
Apache License 2.0
572 stars 102 forks source link

OnlineIndexerIndexFromIndexTest.testIndexFromIndexPersistentContinueRebuildWhenTypeStampChange timeout failures #1267

Closed alecgrieser closed 3 years ago

alecgrieser commented 3 years ago

I've seen at least once during a PRB run that the test OnlineIndexerIndexFromIndexTest.testIndexFromIndexPersistentContinueRebuildWhenTypeStampChange has failed. The error message isn't particularly helpful, with it essentially saying that there's some kind of timeout. Here's the stack trace:

com.apple.foundationdb.record.RecordCoreException
    at com.apple.foundationdb.record.provider.foundationdb.FDBExceptions.wrapException(FDBExceptions.java:200)
    at com.apple.foundationdb.record.provider.foundationdb.FDBDatabase.lambda$new$0(FDBDatabase.java:167)
    at com.apple.foundationdb.record.provider.foundationdb.FDBDatabase.asyncToSync(FDBDatabase.java:1037)
    at com.apple.foundationdb.record.provider.foundationdb.FDBDatabaseRunnerImpl.asyncToSync(FDBDatabaseRunnerImpl.java:340)
    at com.apple.foundationdb.record.provider.foundationdb.OnlineIndexer.asyncToSync(OnlineIndexer.java:790)
    at com.apple.foundationdb.record.provider.foundationdb.OnlineIndexer.buildIndex(OnlineIndexer.java:714)
    at com.apple.foundationdb.record.provider.foundationdb.OnlineIndexerIndexFromIndexTest.buildSrcIndex(OnlineIndexerIndexFromIndexTest.java:71)
    at com.apple.foundationdb.record.provider.foundationdb.OnlineIndexerIndexFromIndexTest.testIndexFromIndexPersistentContinueRebuildWhenTypeStampChange(OnlineIndexerIndexFromIndexTest.java:595)
Caused by: java.util.concurrent.TimeoutException
    at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
    at com.apple.foundationdb.record.provider.foundationdb.FDBDatabase.asyncToSync(FDBDatabase.java:1028)
    ... 95 more

Unclear if this is really the test or if the CI environment can get overwhelmed, though

alecgrieser commented 3 years ago

It's a little concerning that the error message appears not to have a real message associated with it ¯\_(ツ)_/¯

scgray commented 3 years ago

Another similar issue from https://fdb-record-layer-ci.foundationdb.org/job/prb-proto3/2758/

19:17:23 OnlineIndexerIndexFromIndexTest > testIndexFromIndexSrcVersionModifiedWithFallback() FAILED
19:17:23     com.apple.foundationdb.record.RecordCoreException
19:17:23         at com.apple.foundationdb.record.provider.foundationdb.FDBExceptions.wrapException(FDBExceptions.java:200)
19:17:23         at com.apple.foundationdb.record.provider.foundationdb.FDBDatabase.lambda$new$0(FDBDatabase.java:169)
19:17:23         at com.apple.foundationdb.record.provider.foundationdb.FDBDatabase.asyncToSync(FDBDatabase.java:1082)
19:17:23         at com.apple.foundationdb.record.provider.foundationdb.FDBDatabaseRunnerImpl.asyncToSync(FDBDatabaseRunnerImpl.java:340)
19:17:23         at com.apple.foundationdb.record.provider.foundationdb.OnlineIndexer.asyncToSync(OnlineIndexer.java:792)
19:17:23         at com.apple.foundationdb.record.provider.foundationdb.OnlineIndexer.buildIndex(OnlineIndexer.java:713)
19:17:23         at com.apple.foundationdb.record.provider.foundationdb.OnlineIndexerIndexFromIndexTest.buildSrcIndex(OnlineIndexerIndexFromIndexTest.java:72)
19:17:23         at com.apple.foundationdb.record.provider.foundationdb.OnlineIndexerIndexFromIndexTest.testIndexFromIndexSrcVersionModifiedWithFallback(OnlineIndexerIndexFromIndexTest.java:754)
19:17:23 
19:17:23         Caused by:
19:17:23         java.util.concurrent.TimeoutException
19:17:23             at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
19:17:23             at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
19:17:23             at com.apple.foundationdb.record.provider.foundationdb.FDBDatabase.asyncToSync(FDBDatabase.java:1073)
19:17:23             ... 5 more
19:17:23