Decouple MQTT

[nicolemah99]

Implement changes to the archive and data pipelines to streamline the rsync process and handle the MQTT message system on the DAS.

Changes Required

Archive Pipeline

• Remove MQTT
  • [x] Remove all MQTT-related code and dependencies from the archive pipeline.
• Update Archive Configuration
  • [x] Remove MQTT configurations.
  • [x] Simplify the configuration to include only source and destination paths.
  • [x] Add functionality to specify multiple source and destination directories in the config
• [x] Move logic into single file archive_pipeline.sh

Data Pipeline

• Create Message File
  • [x] Generate a .message file with a JSON payload for each event.
  • [x] Update event_id to id.
  • [x] Add a new field date_type or type
  • Example: 1502.zip.message
• Test Default/Override Credentials
  • [x] Ensure the system correctly handles both default and overridden credentials during the data pipeline operations.

    Other Changes

• Document --exclude 'working' for Rsync
  • [x] Update the documentation to include instructions for using the --exclude 'working' option with rsync.