Implement Cleanup Script for Event Files and Working Directories

[nicolemah99]

Issue Summary: Create a cleanup script that systematically removes outdated event files and temporary working directories. This script should be configurable, allowing users to specify how long files should be retained and which directories to clean up.

Objectives:

1. Integrate Cleanup into Archive Process:

   • Implement the cleanup process as part of the archive pipeline, with an option to enable or disable it via a configuration setting.

2. Configurable Retention Period:

   • Allow the user to specify how long files should be kept. The default retention period should be 30 days.

3. Clean Up Old Event Zip Files:

   • Remove zipped event files that are older than the specified retention period.

4. Clean Up Working Directories:

   • Remove old files and directories within the working directory, including any .incomplete and temporary files or folders.

Proposed Solution:

1. Configuration Options:
   • Add a configuration setting to enable or disable the cleanup process.
   • Add a setting to specify the retention period (default: 30 days).
2. Cleanup Script Details:
   • Script Location: camio-meter-streams/cli_meter/cleanup.sh
   • Script Functionality:
   • Check for enabled cleanup in the configuration.
     • Identify and remove event zip files older than the retention period.
     • Identify and clean up files and directories in the working directory older than the retention period.
     • Remove any .incomplete and temporary files/folders.
3. Integrate with Archive Pipeline:
   • Ensure the cleanup script runs at the end of the archive pipeline if enabled.

Example Config:

enable_cleanup: true
  retention_period_days: 30  # Default retention period

Tasks:

• [x] Add configuration options for cleanup enablement and retention period.
• [x] Create cleanup.sh script to remove old files and directories.
• [x] Integrate cleanup.sh into the archive pipeline.
• [x] Test the cleanup functionality with various scenarios. Update documentation to include information on the cleanup process and configuration.