search

MrPowers / mack

Delta Lake helper methods in PySpark
https://mrpowers.github.io/mack/
MIT License
286 stars 42 forks source link

Add unit tests for constraint_append not null constraints #100

Open MrPowers opened 1 year ago

MrPowers commented 1 year ago

Make sure the constraint_append function works well for NOT NULL constraints in the test suite.

See this blog post for more on NOT NULL constraints: https://delta.io/blog/2022-11-21-delta-lake-contraints-check/

cs23andris commented 1 year ago

I'd be happy to add these tests, can you assign it to me please?

robertkossendey commented 1 year ago

@cs23andris Great, assigned you!

cs23andris commented 1 year ago

Unfortunately, NOT NULL constraints are not checked in the current implementation of the constraint_append function, because they're not stored as delta table properties but with the table schema. I think it'd be nice to extend the constraint_append to check for the nullability of columns as well. Can I proceed with that in this issue?

robertkossendey commented 1 year ago

Sounds good and thank you @cs23andris, feel free to work on this! :)

MrPowers commented 1 year ago

Awesome, glad we have unit tests to check the edge cases. Thanks for working on this one @cs23andris.