Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
Having a streaming option for the FileLoader would be beneficial during UT's or for people that are not on Databricks and can't make use of the Databricks proprietary Autoloader.
How Has This Been Tested?
UT's
On Databricks
Types of changes
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
Description
Add streaming option to FileLoader. Default behaviour is batch.
Related Issue
Issue 50
Motivation and Context
Having a
streamingoption for theFileLoaderwould be beneficial during UT's or for people that are not on Databricks and can't make use of the Databricks proprietaryAutoloader.How Has This Been Tested?
Types of changes
Checklist: