delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.64k stars 1.71k forks source link

[BUG] OPTIMIZE command has always 0 end time #1632

Open jaceklaskowski opened 1 year ago

jaceklaskowski commented 1 year ago

Bug

Describe the problem

endTimeMs of OPTIMIZE command (OptimizeStats) is always 0.

Steps to reproduce

// Create a delta table
spark.range(10e4.toLong)
  .repartitionByRange(3, $"id" % 10)
  .write
  .format("delta")
  .save("/tmp/numbers")

// Execute OPTIMIZE
val optimizeMetrics = sql("OPTIMIZE delta.`/tmp/numbers`")

// Print out the metrics
scala> optimizeMetrics.select("metrics.endTimeMs").show
+---------+
|endTimeMs|
+---------+
|        0|
+---------+

Environment information

Willingness to contribute

The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?

allisonport-db commented 1 year ago

Thanks for reporting. Can I assign it to you to fix?