Feature request: Clean Up and Expand `keep_intermediate` Flag in Pipeline

[berntpopp]

Description:

The current keep_intermediate flag in the pipeline is intended to control whether intermediate files are retained after the pipeline execution. However, it does not currently cover all intermediate files generated throughout the pipeline. This can lead to unnecessary disk space usage and potential confusion for users expecting a clean output folder.

Expected Behavior:

• When keep_intermediate is set to false, all intermediate files generated during the pipeline execution should be removed.
• When keep_intermediate is set to true, all intermediate files should be retained as they currently are.

Tasks:

1. Identify All Intermediate Files: Review the entire pipeline to identify all intermediate files generated at different stages.
2. Expand the keep_intermediate Logic:
   • Modify the pipeline to ensure that all identified intermediate files are deleted when keep_intermediate is set to false.
   • Ensure that no essential output files (final results) are removed in the process.
3. Update Documentation:
   • Document the behavior of the keep_intermediate flag, listing the types of files that will be removed or retained based on the flag's value.
   • Provide examples and guidance for users on when to use this flag.

Acceptance Criteria:

• The pipeline behaves as expected when keep_intermediate is set to both true and false.
• All unnecessary intermediate files are removed when keep_intermediate is false.
• The code is reviewed and tested to ensure no critical files are inadvertently deleted.
• Documentation is updated to reflect the changes and guide users on how to use the keep_intermediate flag.

Additional Information:

• This issue is part of an effort to improve pipeline efficiency and user experience by ensuring only the necessary files are retained after execution.