We noticed that the method add_partitions_if_not_exists has a silent bug (from the Hive Metastore Server): if some partition of partition_list does not exist, the entire list will not be added, and no error will be raised.
We fix this in this PR adding each partition individually in a loop.
Also, we created a new method to be used in the place of this one if the exception is desired.
What? :wrench:
Make the add_partitions_if_not_exists add each partition individually through a for loop.
Add the method add_partitions_to_table that will raise an error if some partition already exists and interrupt the execution.
Type of change :file_cabinet:
[X] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[X] This change requires a documentation update
How everything was tested? :straight_ruler:
Unit test + testing with a Hive Metastore Server
Checklist :memo:
[X] I have added labels to distinguish the type of pull request.
[X] My code follows the style guidelines of this project (docstrings, type hinting and linter compliance);
[ ] I have performed a self-review of my own code;
[X] I have made corresponding changes to the documentation;
[X] I have added tests that prove my fix is effective or that my feature works;
[X] I have made sure that new and existing unit tests pass locally with my changes;
Why? :open_book:
We noticed that the method
add_partitions_if_not_exists
has a silent bug (from the Hive Metastore Server): if some partition ofpartition_list
does not exist, the entire list will not be added, and no error will be raised. We fix this in this PR adding each partition individually in a loop. Also, we created a new method to be used in the place of this one if the exception is desired.What? :wrench:
Make the
add_partitions_if_not_exists
add each partition individually through a for loop. Add the methodadd_partitions_to_table
that will raise an error if some partition already exists and interrupt the execution.Type of change :file_cabinet:
How everything was tested? :straight_ruler:
Unit test + testing with a Hive Metastore Server
Checklist :memo: