jepsen-io / jepsen

A framework for distributed systems verification, with fault injection
6.68k stars 711 forks source link

Improved bank workload proposal #556

Open qvad opened 1 year ago

qvad commented 1 year ago

Original bank workload includes only UPDATE operations, while there is a way to create a bank workload with DELETE and INSERTS. It's needed when we want to test tombstones feature in DBs, e.g.

There is 2 suggestions how it can be implemented.

  1. W/o creating test on Jepsen side. Currently implemented for Yugabyte DB here:

Create pool of contention keys, for example: Scenario has 10 rows in table [0-9], first 5 keys is for only UPDATE operation, other 5 is for random INSERT and DELETEs.

INSERT:

update balance = balance - AMOUNT where key = 0;
insert balance = AMOUNT where key = 5;

DELETE:

select AMOUNT where key = 5;
delete where key = 5;
update balance = balance + AMOUNT where key = 0;

This can be implemented w/o many ugly code on the client side.

  1. Add new test scenario:

There might be few different implementations, I suggest to add a scenario with rolling key segments: Starting segment of keys from [0-9] -> [1-10] -> [2-11] -> [3-11] -> [3-12] -> ... This approach allows us to add INSERT-DELETE operation combinations, while the first one (with contention keys) not allow it.

What do you think?

aphyr commented 1 year ago

I think that's a great idea! One thing you could do is to modify the generator so that it keeps track of likely balances (use the new update function for this!) and have it occasionally generate transfers which would (most likely) empty an account. Then clients can choose to delete empty accounts and create new accounts where none exists right now. This has the advantage that the existing checker, visualizations, and all the tests that currently use the bank namespace will work without any changes.

Alternatively you could create a new :f for destroying accounts--might be simpler to implement the generator, but you'd have to add support (or put it behind some sort of configuration option) to existing tests, and teach the checkers how to interpret that new function. I think the advantage here is that under heavy contention (and that's the interesting case) predicting the balance of an account is tough, so the "just do transfers that empty accounts" tactic might not actually generate deletions as often as you might like.

qvad commented 1 year ago

Thanks for a feedback, @aphyr! I'm not sure that predicting empty accounts may work well for all systems, will try to implement scenario with new:f.