search

ept / hermitage

What are the differences between the transaction isolation levels in databases? This is a suite of test cases which differentiate isolation levels.
http://martin.kleppmann.com/2014/11/25/hermitage-testing-the-i-in-acid.html
2.45k stars 182 forks source link

Coordinate (and merge?) these efforts with Jepsen #7

Closed dumblob closed 5 years ago

dumblob commented 5 years ago

Jepsen does very good job in finding safety issues in databases (most well known databases failed their safety tests pretty hard).

It might make sense to somehow cooperate with Jepsen or even merge your efforts?

ept commented 5 years ago

Thanks for your suggestion. I'm a big fan of Jepsen, and I gather Kyle sometimes refers to Hermitage as well. However, while the two projects support each other, I think they are quite different in many ways, and thus I think it makes sense to keep them separate.

miparnisari commented 12 months ago

@ept as someone who is unfamiliar with Jepsen 😅 do you mind documenting what is the difference with it? Thanks!

ept commented 11 months ago

@miparnisari Sure. Jepsen is a test framework that automatically generates lots of inputs to a system, and concurrently controls the network and the clocks on the nodes to try to cause trouble. It then checks whether the results are correct. Hermitage is much simpler – it doesn't run tests automatically, it's just some queries that you can manually copy and paste into a terminal window. As such, Hermitage isn't really software, it's more like documentation. In principle, one could write an automated test runner based on the examples in this repo, but that would be a separate project. I like the fact that Hermitage tests are run manually, because the process of running them manually helps us build an intuition for how concurrency control mechanisms work.

(The FoundationDB tests are an exception – they are run by a shell script because the transaction would time out if the queries were copied and pasted by hand.)