Transactions issued on different nodes seems affected by lost updates

Hi. If I understand it correctly, transactions issued on different nodes in a cluster should be protected from lost updates, as claimed in https://galeracluster.com/library/training/tutorials/supporting-transaction-isolation-levels.html. However, we have observed this anomaly in one of our tests.

We have set up our tests with the following configuration:

We run a cluster of two nodes using docker-compose (configuration). The versions are 10.7.3 for mariadb and 26.4.11 for galera. We are using a simple schema. There is only one table, each row containing only a key and a value:

CREATE TABLE IF NOT EXISTS dbcop.variables (var BIGINT(64) UNSIGNED NOT NULL PRIMARY KEY, val BIGINT(64) UNSIGNED NOT NULL)

We have one client for each node. First, we create the rows and initialize their values to 0. Then, the clients run a set of transactions generated by our test generator. The code for the clients can be found at galera.rs and cluster.rs. The values written by the clients are unique.

In one of our tests, there is a single row with key=0. We observed the following execution:

T{n} -> transaction n
!T{m} -> failed to commit
R(0): {x} -> read (var = 0, val = x)
--- Node 1               --- Node 2
--- T1                   --- !T6
W(0): 1                  R(0): 0
W(0): 2                  W(0): 8
--- T2                   --- !T7
W(0): 3                  R(0): 2
W(0): 4                  W(0): 9
--- T3                   --- T8
R(0): 4                  R(0): 4
W(0): 5                  W(0):10
--- T4                   --- T9
R(0): 5                  W(0):11
R(0): 5                  W(0):12
--- !T5                  --- T10
W(0): 6                  W(0):13
W(0): 7                  W(0):14

In this history, both T3 and T8 reads val=4. Then they proceed to overwrite this value and commit successfully. After that, T4 reads val=5, which means the write by T8 is lost.

This behavior seems to contradict the second example given in the documentation linked above, in which two transactions are issued on separate nodes, and one of them fails to commit because they try to update the same row. I can replicate this example when trying to follow it by hand using the test setup, though.

The tests can be run with the following steps:

Run the cluster in docker using the docker-compose.yml file.

Clone and run the test:

git clone https://github.com/amnore/dbcop && cd dbcop
mkdir -p clientops result
cargo run generate -d clientops -h 5 -n 2 -t 5 -e 2 -v 1 # generate transactions
cargo run --example galera 127.0.0.1:3306 127.0.0.1:3307 --dir clientops --out result # run history on cluster

Use verifier to verify the results

git clone https://github.com/amnore/CobraVerifier && cd CobraVerifier
gradle jar
java -jar build/libs/CobraVerifier-0.0.1-SNAPSHOT.jar audit -t dbcop ${path_to_result}/hist-00000/history.bincode
# replace hist-00000 with other folders in ${path_to_result} to verify other histories

When a lost-update is found, the verifier will print something like this:

Conflicts:
Edge: (<(2, 6) -> (1, 2)>,[(RW, 0)])
Edge: (<(1, 2) -> (2, 6)>,[(WW, 0)])
Relevent transactions:
sessionid: 2, id: 6
ops:
READ 0 = 1
WRITE 0 = 6
sessionid: 1, id: 2
ops:
READ 0 = 1
WRITE 0 = 2

There are two transactions in the example. Transaction 6 reads (key=0, val=11) and writes (key=0, val=6). Transaction 2 reads (key=0, val=1) and writes(key=0, val=2).

Print the original history

# in dbcop folder
cargo run -d result/hist-00000

This command will produce outputs like this:

History { params: HistParams { id: 0, n_node: 2, n_variable: 1, n_transaction: 5, n_event: 2 }, info: "Galera", start: 2022-05-17T18:00:25.964453190+08:00, end: 2022-05-17T18:00:25.986436117+08:00, data: [
[[<W(0): 1>, <R(0): 1>], [<R(0): 1>, <W(0): 2>], [<R(0): 2>, !<W(0): 3>], [<R(0): 6>, <W(0): 4>], [<R(0): 4>, <R(0): 4>]], 
[![<W(0): 5>, <R(0): 5>], [<R(0): 1>, <W(0): 6>], [<R(0): 6>, <R(0): 6>], ![<W(0): 7>, <W(0): 8>], [<W(0): 9>, <W(0):10>]]] }

![<W(0): 5>, <R(0): 5>] is a transaction, and the prefix ! means it didn't commit successfully.

In our tests, this phenomenon appears quite often (about 3 times in 10 runs), though the outputs of the verifier may differ.

Is this the expected behavior? Or am I missing something in the docs?

codership / galera

Transactions issued on different nodes seems affected by lost updates #618