[Spark] Ensure commit directory is created for old and new tables - Githubissues

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

https://delta.io

Apache License 2.0

6.96k stars 1.58k forks source link

[Spark] Ensure commit directory is created for old and new tables #3000

Closed sumeet-db closed 1 week ago

sumeet-db commented 2 weeks ago

Which Delta project/connector is this regarding?

[x] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

Remove createLogDirectory and replace all usages with ensureLogDirectoryExists since the latter is optimized for creation rather than existence check now. Rename it createLogDirectoriesIfNotExists
Ensure commit directory is created on existing tables from older releases when manged-commits is enabled.

How was this patch tested?

UTs

Does this PR introduce any user-facing changes?

No

tdas commented 2 weeks ago

Yooohoooo. THREE THOUSAND!!