Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
The exists function of the DeltaTableStep should notify the user about the non existence of the table with a different log level depending on the value of the create_if_not_exists boolean flag (whereas now it is logging always with level error).
If the flag is True than the log level is set to info. If the flag is False than the log level is set to error.
A possible alternative is to set the log level always to info or warning but this solution proposed already by the reporter looks more informative.
Related Issue
34
Motivation and Context
Improving logging by making more informative and adeguate to the use case.
How Has This Been Tested?
By adding unit tests that verify the log message contains a certain string.
Screenshots (if appropriate):
Types of changes
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
… run black
Description
The
existsfunction of theDeltaTableStepshould notify the user about the non existence of the table with a different log level depending on the value of thecreate_if_not_existsboolean flag (whereas now it is logging always with level error). If the flag is True than the log level is set to info. If the flag is False than the log level is set to error.A possible alternative is to set the log level always to info or warning but this solution proposed already by the reporter looks more informative.
Related Issue
34
Motivation and Context
Improving logging by making more informative and adeguate to the use case.
How Has This Been Tested?
By adding unit tests that verify the log message contains a certain string.
Screenshots (if appropriate):
Types of changes
Checklist: