We need to enhance our download pipeline to ensure that it only downloads events that are newer than a specified maximum age. This feature will prevent the system from downloading outdated events. The maximum age should be configurable in days.
Tasks
Update Configuration:
Add a new configuration parameter max_age_days to the existing configuration file.
Document the new configuration parameter.
Read Configuration in Data Pipeline:
Update the data_pipeline.sh script to read the max_age_days parameter from the configuration file.
Pass to download script
Pass Max Age to Download Script:
Modify the download.sh script to accept and pass the max_age_days parameter to the get_events.sh script.
Update get_events.sh Script:
Modify the script to calculate the event age and compare it with the max_age_days parameter.
Only include events in the output that are newer than the specified maximum age.
We need to enhance our download pipeline to ensure that it only downloads events that are newer than a specified maximum age. This feature will prevent the system from downloading outdated events. The maximum age should be configurable in days.
Tasks
Update Configuration:
max_age_daysto the existing configuration file.Read Configuration in Data Pipeline:
data_pipeline.shscript to read themax_age_daysparameter from the configuration file.Pass Max Age to Download Script:
download.shscript to accept and pass the max_age_days parameter to the get_events.sh script.Update get_events.sh Script:
max_age_daysparameter.