Wanted: Transaction benchmark scenarios

[philbe]

@tamereldeeb and I are looking for mixed-workload transaction benchmarks that we can use to measure our implementation’s performance. It should have a few types of transactions and be representative of transaction apps that Orleans users want. Do you have any scenarios to suggest?

TPC-C is the most popular transaction benchmark. But each transaction fans out to many grains, which will perform poorly compared to a classical database system, and is probably not representative of transaction apps that Orleans users want. TPC-E is too complicated and again is probably not representative.

We have one good benchmark, indexing, where a transaction updates the grain and the index over the grain’s class. We’re looking for something more complex.

[veikkoeeva]

@philbe I'm not sure if TPC-DS 2.0 could be adapted, since it's geared towards big data and read-heavy loads, while still having some general purpose processing. It looks like the 2.0 version was created specifically because the 1.0 version was cherry-picked by vendors of relational and non-relational, and it looks like it's happening to some extent in 2.0 too (code). The Hadoop ecosystem seem to be gravitating around this.

[philbe]

@veikkoeeva Interesting thought. The good part is that TPC-DS 2.0 mixes updates with queries, which is probably what we want. But off hand, I don't see how to adapt it to meet our needs. The updates in TPC-DS are files, each containing updated rows of a table. AFAIK, they don't tell us the nature of transactions that generated those updated rows, so we would have to guess at the transaction workload that produced those rows.

Someone suggested OLTPBench (https://github.com/oltpbenchmark). It generates load for a lot of benchmarks. Kind of embarrassing I didn't think of that, since I know the authors well. I'd be interested to know if any of their workloads is representative of some Orleans apps.

[veikkoeeva]

I would not credit this idea too much, it's a rather layman note. From this perspective the appeal is that it looks like it is getting attention and maybe people considering actor systems such as Orleans are thinking if they could have a more integrated approach, have a more all-in-one framework instead of separate products. From this perspective, the mention in one of the aforementioned sources

TPC-DS 2.0 requires the system under test (SUT) to continue executing queries and data-maintenance functions with full data access during and after a permanent irrecoverable failure of any single durable medium. TPC-DS 2.0 for the first time mandates a durability test on the SUT during the performance tests. This makes the durability test not only a functionality test, but also a performance test.

becomes even more interesting. This is more of a "marketing speach", though. Now that I think of this, YCSB could be interesting too, but I take a look at OLTP benchmark also.

Maybe the point to extract is that it would be cool to have Orleans benchmarked in some standard way like this without purpose-built big boxes like one sees in the TPC-* benchmarks. Orleans might be chosen in general by other criteria for now, but as it looks like competing in the future in the same arena with some of the Hadoop based offerings, for instance, so it would benefit being measured somehow with the same tests.

<edit: Digging Deeper into YCSB Benchmark with Couchbase Server 4.5 as an example of YCSB benchmark. One could think that what if the clients were Orleans clients and the nodes were silos, especially since in-memory indexes are mentioned several times. Again, lay-mean observations. If there will be a public benchmark, an educational reasoning of the usage and scenarios would be highly appreciated. :)

[ashkan-saeedi-mazdeh]

@philbe Sorry for seeing this late. I can speak of I myself and my scenarios specifically in gaming and can mention workloads which I imagine people will use.

• Updating an index based on a grain's state is the most obvious you mentioned. This is specially useful in Orleans since all grains virtually always exist and supervision trees are not the recommended/required practice around here so for (say managing leaderboards) we don't have a grain which manages the hierarchy of leaderboard grains or at least we want to be able to avoid it since the programming model can allow that easily with an index.
• Trading in games or other kinds of realtime applications. So changing the state of 2-5 grains altogether or none of them. This is both valid for traind in a game environment which something from my inventory is moved to yours or in a betting environment where server gets say 50 coins from each of us (either both or none should be successful) and at the end of the match rewards the winner with 100 of coins. Also in a stack overflow like website if I put 100 of my reputations on a question then the one who answers should get them and the value should be reduced from my points and added to his.
• Managing graph like structures like friendships. Imagine a friendship system like facebook's which is bidirectional and not twitter's follow system which is unidirectional. When you accept my friendship request then 3 operations should be in the transaction, updating of invitation, updates of friends list of both of us. A match making system can be kinda similar however that doesn't use storage most of the times but still if my request of finding a match is considered complete by finding a match, then the game should be created, my opponent and I both should get that match and change our state to the one which represents found a match.

All this sayd I would say most of these are implementable without transactions but still the scenarios I would see for transacitons in Orleans are only these small transactions spanning a few grains most of the times. Based on what other users are implementing in Orleans, purchase processing, fraud detection, online betting/gambling/gaming systems and ... I think many of the cases are answered by this probably less heavy kind of transactions. Foregive my examples of gaming but to me games are one of the most useful scenarios which lend themselves well into actor systems (after all actors are proposed for multi-agent simulations at first as well and Orleans is being maintained by the glorious Microsoft Game Studios) for a reason.

[philbe]

@ashkan-saeedi-mazdeh, good examples! They're consistent with our intuition that for Orleans, transactions will span a relatively small number of grains (unlike TPC-C). In particular, others have asked us for transactions to support trading scenarios.

[ashkan-saeedi-mazdeh]

@philbe When the time comes, You can count me for testing in terms of usability testing or finding early bugs/... as well. I have an Orleans based system which we'll put in production for beta testing in 3-5 days and there are places which I can add transactions and I have good test coverage so hopefully cn find issues as well.

[philbe]

@ashkan-saeedi-mazdeh Great! Consider yourself hired. :-)

[AshkanSaeedi]

@philbe Awesome, Hopefully soon I'll report from active duty :)