Add more tests and logic check the skipped_stats function

mrpowers-io / levi

Delta Lake helper methods. No Spark dependency.

MIT License

21 stars 8 forks source link

Add more tests and logic check the skipped_stats function #9

Open MrPowers opened 1 year ago

MrPowers commented 1 year ago

The skipped_stats function (and the filters_to_sql / filter_to_sql queries it rely on) should be better tested. There are probably corner cases / bugs in the function.

The function could also use a logic check from a pair of fresh eyes.

brayanjuls commented 1 year ago

@MrPowers - hey! I would like to work on this.

MrPowers commented 1 year ago

Awesome, thanks @brayanjuls

brayanjuls commented 1 year ago

@MrPowers - working on this issue I just realized we can do the same in delta scala using DeltaLog..snapshot.filesWithStatsForScan, after finishing with this I will take a deeper look at it.