Multiple operations, ~10, using the Java client crash the server.

nurturenature commented 2 years ago

Hi,

I am creating a Jepsen test for TigerBeetle and am using the Java client. (Jepsen is written in Clojure so it's a good fit.)

The tests can only generate a small number of operations, ~10, before the database crashes:

Jepsen starting  /root/tigerbeetle/tigerbeetle start --addresses=192.168.122.101:3000 0_0.tigerbeetle
info(io): opening "0_0.tigerbeetle"...
info(main): 0: cluster=0: listening on 192.168.122.101:3000
info(message_bus): connection from client 317247975010753124847524919045321411137
error(storage): short read: buffer.len=706953216 offset=65536 bytes_read=0
thread 840 panic: data file inode size was truncated or corrupted

Most common error is read failure, have also gotten unreachable code.

In a multi replica environment the tests will continue to crash replicas until < quorum remains.

Does not matter the type of operation:

create/lookup accounts
create transfers

Clients become unresponsive during, after failure.

I have updated the integration tests to loop over account creation/lookup to demonstrate the bug.

eatonphil commented 2 years ago

Hey! This is really fantastic, thanks for hooking this up!

The storage layer was just merged recently but is not yet released or production ready. We're working through a bunch of known issues right now so this is really great as another source of testing for us! Thank you!

nurturenature commented 2 years ago

Thanks for the update!

The associated PR comments indicate that the API limiting is intentional for this stage of development, so will close this issue.

Am looking forward to experimenting with Jepsen tests as things progress.

tigerbeetle / tigerbeetle-java

Multiple operations, ~10, using the Java client crash the server. #9