Design and implement the data retention strategy

[ydaponte]

Parent #578 Dependency #579

Pre-requirements:

• Having a data lake hierarchy to test.
• To integrate with the final sample, the e2e sample needs to be deployed.

Data will be deleted based on the field tpep_pickup_datetime. The goal would be to create a deletion notebook or sql scripts that implements:

• A delete of an entire month (partition) based on the configuration file setting
• A delete based on the tpep_pickup_datetime that no longer needs to be retained based on the conf file setting

Success criteria

• 2 notebooks or sql scripts that implements each use case
• Calling the second parametrized notebook/script from the pipeline