Provide insights on the columns with stats collected

mrpowers-io / levi

Delta Lake helper methods. No Spark dependency.

MIT License

21 stars 8 forks source link

Provide insights on the columns with stats collected #6

Open MrPowers opened 1 year ago

MrPowers commented 1 year ago

Some Delta implementations only collect stats on certain columns.

Here are some APIs that will help users debug their questions:

what columns are stats being collected for
for a given column, are stats being collected in all the files?
what columns have stats being collected for some files, but not other files

The get_add_actions API should make this possible.

robertkossendey commented 1 year ago

Great idea, do we want to return the result as an object, so the user can work with the output programatically, or just print it?

MrPowers commented 1 year ago

@robertkossendey - I generally prefer outputting results programmatically, so users can use the functions in programs. There are some exceptions, but I think computer-readable output (that's also human friendly) is usually the best goal.

robertkossendey commented 1 year ago

@MrPowers you can assign me here, I'd like to work on this.