[Spark] Fix inCommitTimestamp deserialization for very small timestamps

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Apache License 2.0

7.17k stars 1.62k forks source link

[Spark] Fix inCommitTimestamp deserialization for very small timestamps #3077

Closed dhruvarya-db closed 1 month ago

dhruvarya-db commented 1 month ago

Which Delta project/connector is this regarding?

[X] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

Follow up for https://github.com/delta-io/delta/commit/2ef3a2ae887e4d79c1469c60c54b3ec993b97db0 . There is another site where we don't specify the right way to serialize inCommitTimestamp which will cause failures when deserializing very small timestamps.