Describe the solution you'd like
Exporting a huge table into multiple smaller files allows users to effectively parallelise the transformation afterwards, using tools like Spark and Beam.
[ ] Test case with valid end-to-end transfer from a Table to multiple files
[ ] All checks and tests in the CI should pass
[ ] Unit tests (90% code coverage or more)
[ ] Integration tests (if the feature relates to a new database or external service)
[ ] Example DAG
[ ] Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
[ ] Exception handling in case of errors
[ ] Logging (are we exposing useful information to the user? e.g. source and destination)
[ ] Improve the documentation (README, Sphinx, and any other relevant)
Please describe the feature you'd like to see
We can also have multiple file — good to have not a must-have feature
We can assume a default file_size_threshold, once reached we can split data into multiple file.
Describe the solution you'd like Exporting a huge table into multiple smaller files allows users to effectively parallelise the transformation afterwards, using tools like Spark and Beam.
Additional context More details at: notion doc
Acceptance Criteria