An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
As of now, Vacuum dry run mode returns a dataframe containing a list of first 1000 files to be deleted. Instead, it's more useful to return all the eligible files to be deleted along with the size of these files.
Motivation
Today, users run Vacuum in dry run mode to understand the eligible files and the size of the data that is going to be deleted. By returning only a sample of 1000 files, users are not able to achieve their objective.
Further details
Willingness to contribute
The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?
[ ] Yes. I can contribute this feature independently.
[X] Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
[ ] No. I cannot contribute this feature at this time.
Feature request
Which Delta project/connector is this regarding?
Overview
As of now, Vacuum dry run mode returns a dataframe containing a list of first 1000 files to be deleted. Instead, it's more useful to return all the eligible files to be deleted along with the size of these files.
Motivation
Today, users run Vacuum in dry run mode to understand the eligible files and the size of the data that is going to be deleted. By returning only a sample of 1000 files, users are not able to achieve their objective.
Further details
Willingness to contribute
The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?