search

apple / foundationdb

FoundationDB - the open source, distributed, transactional key-value store
https://apple.github.io/foundationdb/
Apache License 2.0
14.38k stars 1.3k forks source link

SwizzledApiCorrectness test failure on master #4697

Closed sfc-gh-jfu closed 2 years ago

sfc-gh-jfu commented 3 years ago

A failed assertion in the DiskQueue actor, preceded by a large amount of SimulatedFDBDTerminated: A disk IO operation failed to complete in a timely manner messages before failing the assertion.

<InternalError Severity="40" DateTime="2021-04-21T05:23:18Z" Error="internal_error" ErrorDescription="An internal error occurred" ErrorCode="4100" FailedAssertion="len == sizeof(Page)" File="/mnt/jenkins/home/jenkins/workspace/FoundationDB-OSFDB-Build/jenkins/foundationdb/fdbserver/DiskQueue.actor.cpp" Line="697" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x266befc 0x266b490 0x266b761 0x25d5c2f 0xccd707 0xccd789 0x25501f8 0x2530330 0x25501f8 0x255437b 0xcb0ec0 0x252e318 0x254ac80 0x252e41d 0x254adac 0xcb0ec0 0x261152b 0x254aee3 0x82f25f 0x7f3f4176f555" LogGroup="default" Roles="LR,SS,TL"/>

Ensemble: 20210421-041609-nightly_correctness_master-42837d4969a2bb47 Test Details:RandomSeed="605750695" SourceVersion="58d1a6f9c0eed8742e5200f013426f41bb2ba908" Time="1618982540" BuggifyEnabled="1" DeterminismCheck="0" OldBinary="" TestFile="/tmp/joshua_agent/ensembles/20210421-041609-nightly_correctness_master-42837d4969a2bb47/tests/slow/SwizzledApiCorrectness.toml" Master SHA: https://github.com/apple/foundationdb/commit/58d1a6f9c0eed8742e5200f013426f41bb2ba908

sfc-gh-ljoswiak commented 3 years ago

Another SwizzledApiCorrectness failure on commit 061afda2ec108b9343a3306a5b07824ec41eb28f (possibly unrelated):

$ ./bin/fdbserver -r simulation -f ../foundationdb/tests/slow/SwizzledApiCorrectness.toml --seed 903160801 --buggify on
<Event Severity="40" ErrorKind="Unset" Time="20313.611141" DateTime="2021-05-14T18:59:39Z" Type="TestFailure" Machine="0.0.0.0:0" ID="0000000000000000" Error="timed_out" ErrorDescription="Operation timed out" ErrorCode="1004" Reason="Unable to perform consistency check" Backtrace="addr2line -e fdbserver.debug -p -C -f -i 0x282f50c 0x28305c0 0x28309a1 0x14fe9eb 0x14fecd5 0x8c54b9 0xfb18a3 0xd15d50 0x26e02d8 0x26fdaa0 0x26e03dd 0x26fdc3f 0xd15d50 0x27cef83 0x26fddf2 0x86c729 0x7f53e3fa51a2" LogGroup="default" />