Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
Is your feature request related to a problem? Please describe.
Having a streaming option for the FileLoader would be beneficial during UT's or for people that are not on Databricks and can't make use of the Databricks proprietary Autoloader.
Describe the solution you'd like
A streaming option like the DeltaTableReader has as seen here
Describe alternatives you've considered
AutoLoader is an alternative but isn't present outside of Databricks ...
Is your feature request related to a problem? Please describe.
Having a streaming option for the
FileLoaderwould be beneficial during UT's or for people that are not on Databricks and can't make use of the Databricks proprietary Autoloader.Describe the solution you'd like
A
streamingoption like theDeltaTableReaderhas as seen hereDescribe alternatives you've considered
AutoLoader is an alternative but isn't present outside of Databricks ...