search

TPC-Council / HammerDB

HammerDB Database Load Testing and Benchmarking Tool
http://www.hammerdb.com
GNU General Public License v3.0
590 stars 119 forks source link

Add TPC-E as a workload #1

Closed abondvt89 closed 10 months ago

abondvt89 commented 5 years ago

Add TPC-E as a workload

ghost commented 3 years ago

Is TPC-E in the works or still in the planning phase? Based on the published benchmarks my sense is that Oracle and IBM will not be publishing any anytime soon. Does this inhibit the team from moving forward with TPC-E?

billramo commented 3 years ago

For what it's worth, supporting TPC-E doesn't make sense based on feedback from benchmark experts from database vendor and hardware vendor teams. Only SQL Server has published results for the benchmark. Here is a link to a technical paper that outlines the issues with TPC-E - From A to E: Analyzing TPC’s OLTP Benchmarks. In essence, the TPC-E benchmark doesn't take full advantage of the CPU, Memory, and Disk resources as you scale the benchmark up.

ghost commented 3 years ago

That’s good to know. I haven’t been able to get EGenDriver to work so I’ve been looking for an easier path.

On Tue, Nov 10, 2020 at 20:26 Bill Ramos notifications@github.com wrote:

For what it's worth, supporting TPC-E doesn't make sense based on feedback from benchmark experts from database vendor and hardware vendor teams. Only SQL Server has published results for the benchmark. Here is a link to a technical paper that outlines the issues with TPC-E - From A to E: Analyzing TPC’s OLTP Benchmarks https://openproceedings.org/2013/conf/edbt/TozunPKJA13.pdfl. In essence, the TPC-E benchmark doesn't take full advantage of the CPU, Memory, and Disk resources as you scale the benchmark up.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/TPC-Council/HammerDB/issues/1#issuecomment-725087536, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIHM7ORLZRMSIIWKFCZY6XDSPHY35ANCNFSM4GIFS3BA .

-- Stephen Moon (615) 920-2415

sm-shaw commented 3 years ago

Many thanks Bill for the excellent insights and the link to the paper. Yes, currently TPC-E is not planned for consideration until both Oracle and Db2 have published fully audited TPC-E benchmarks. What is in planning is to add the Hybrid CH benchmark to HammerDB #123. This will not be immediate, firstly v4.0 will be released and additional updates following this with the CH benchmark planned for the next major release.

billramo commented 3 years ago

Hi Steve, It's great to hear you are looking to impliment the Hybrid CH benchmark. Microsoft created a version using their Benchcraft tool kit to run the TPC-C portion and PowerShell to run the TPC-H queries. The database tables contained relationally consistent data for both benchmarks.

This test was "brutal" on all database engines. In 2016, I worked with Microsoft to port their SQL Server version to Oracle, DB2, SAP HANA, Sybase ASE, PostgreSQL, MySQL, SQL Server, Azure SQL Database, and MemSQL. The version we created supported Azure and AWS clouds. Jamie Reding was our product team sponsor at Microsoft. Maybe he can provide you with the data generator algorithm for the benchmark.

I'd like to help in this adventure. Regards, Bill

sm-shaw commented 10 months ago

As discussed at TPC-OSS meeting 14th Nov. This issue was transferred from the original HammerDB sourceforge project to the TPC-Council on GitHub and having not been actioned in the interim period is now considered stale. If there is a wish to contribute this enhancement, then this issue can be reopened and self-assigned.