Add a configuration file to the sample

[ydaponte]

Dependency #577

The configuration file will be responsible to drive the following points:

• Automate what data needs to be ingested
• Automate what data needs to be updated ( means ingest again and re-create partitioned view)
• Automate data deletion based on data retention settings
• Automate permissions (ACLs) definition in the data lake hierarchy
• [Stretch Goal] - Automate permissions on the columns of the partitioned views

Sub-Task of this User Story can be summarized as:

• Define configuration file structure
• Read and write into the configuration file from the pipeline
• Upload configuration file as part of the sample deployment
• Modify pipeline to ingest new data or update already ingested data based on configuration file
• Automate ACL definitions based on configuration file
• Define a retention strategy, for example: Data will be deleted based on the field tpep_pickup_datetime
• Create a deletion notebook that implements:

• A delete of an entire month (partition) based on the configuration file setting
• A delete based on the tpep_pickup_datetime that no longer needs to be retained based on the conf file setting

Success Criteria

• Modified and tested setup pipeline to ingest and modify data based on the configuration file
• Modified and tested setup pipeline to create and/or re-create partitioned views based on the configuration file
• Modified and tested setup pipeline to delete data and corresponding partitioned views that no longer needs to be retained
• Modified and tested setup pipeline to define ACL permissions based on the configuration file
• [Stretch Goal] - Modified and tested setup pipeline to automate the column permissions definitions