Database vendors should change to use the “DeWitt Embrace Clause”: if a competitor or vendor benchmarks Databricks or instructs a third party to do so, this new provision invalidates the vendor’s own DeWitt Clause if there’s any, to allow us to benchmark them and explicitly name them in the benchmark.
Azure, AWS and Databricks have used the “DeWitt Embrace Clause”.
Snowflake doesn't want to engage in benchmarking wars and claims divorced from real-world experiences as people did twenty years ago.
The price of Snowflake measured by Databricks is about 5.7x higher than measured by Snowflake. The truth is that Snowflake has roughly the same performance and price at $267 as Databricks SQL at $242.
Everyone can verify the result announced by Snowflake easily by taking a few mouse clicks without any prior experience.
Snowflake changed to accept the “DeWitt Embrace Clause”.
Using spot instances to run Databricks (as most customers use) brings the price down to $146.
Databricks used the dataset that pre-loaded by Snowflake and found that Snowflake costs 4,025 seconds, which is similar to 3,760 seconds that Snowflake claimed.
After loading the official TPC-DS dataset into Snowflake, Databricks found that the time consumed by the power test took from 7,276 to 10,085 seconds, which was at least 1.9x longer than what Snowflake reported in their blog.
So Databricks assumed that Snowflake might have done some pre-clustering or pre-partitioning (such as sorting by the combination of fields used in a query) in the pre-loaded dataset.
Databricks engineering team were not allowed to apply any optimizations that would require deep understanding of the dataset or queries.
Snowflake should verify their results with the official TPC council.
Databricks: Databricks Sets Official Data Warehousing Performance Record
Databricks: Eliminating the Anti-competitive DeWitt Clause for Database Benchmarking
Snowflake: Industry Benchmarks and Competing with Integrity
Databricks: Snowflake Claims Similar Price/Performance to Databricks, but Not So Fast!