search

cockroachdb / loadgen

CockroachDB load generators
Apache License 2.0
30 stars 25 forks source link

tpcc: sanity check txn requirements #151

Closed petermattis closed 6 years ago

petermattis commented 6 years ago

The TPC-C spec has sanity checks on the randomized txns (9.2.2.5):

  1. At least 0.9% and at most 1.1% of the New-Order transactions must roll back as a result of an unused item number.
  2. The average number of order-lines per order must be in the range of 9.5 to 10.5 and the number of order- lines per order must be uniformly distributed from 5 to 15 for the New-Order transactions that are submitted to the SUT during the measurement interval.
  3. The number of remote order-lines must be at least 0.95% and at most 1.05% of the number of order-lines that are filled in by the New-Order transactions that are submitted to the SUT during the measurement interval.
  4. The number of remote Payment transactions must be at least 14% and at most 16% of the number of Payment transactions that are submitted to the SUT during the measurement interval.
  5. The number of customer selections by customer last name in the Payment transaction must be at least 57% and at most 63% of the number of Payment transactions that are submitted to the SUT during the measurement interval.
  6. The number of customer selections by customer last name in the Order-Status transaction must be at least 57% and at most 63% of the number of Order-Status transactions that are submitted to the SUT during the measurement interval.

We should add metrics to track the above and output these metrics when the test ends along with an indication of whether we met the requirements.

petermattis commented 6 years ago

@arjunravinarayan This is related to fleshing out the consistency checks and would be another level of sanity checking that we should perform.

rjnn commented 6 years ago

SGTM.

rjnn commented 6 years ago
  1. Verify that no more than 1%, or no more than one (1), whichever is greater, of the Delivery transactions skipped because there were fewer than necessary orders present in the New-Order table. (9.2.1.7)
  2. O_ALL_LOCAL can be used to verify that approximately 10% of all orders contain at least one remote order-line. (9.2.2.9)
  3. Perform distributional checks on S_YTD, S_ORDER_CNT, S_REMOTE_CNT, C_PAYMENT_CNT (also 9.2.2.9).
solongordon commented 6 years ago

This is all set except for the 9.2.2.9 checks. I propose not adding those because a) they're kind of vague and b) the TPC-C spec makes them sound optional ("The auditor can further verify the compliance..."). Any objections?

solongordon commented 6 years ago

Relevant PRs for future reference: