MrPowers / mack

Delta Lake helper methods in PySpark
https://mrpowers.github.io/mack/
MIT License
303 stars 44 forks source link

Possibly make the quarantine_table argument optional in constraint_append #101

Closed MrPowers closed 1 year ago

MrPowers commented 1 year ago

constraint_append currently requires a quarantine_table. We may want to allow users to set quarantine_table to None and just throw out the rows that don't match the constraints.

There are probably some users that want to write the bad data to a quarantine table and others that just want to throw out the bad data. Thoughts?

danielbeach commented 1 year ago

@MrPowers I can work on this one if you want.

robertkossendey commented 1 year ago

@danielbeach thank you very much, assigned you! :)