User arguments to retain intermediate files

[delucchi-cmu]

Closes #297

Solution Description

Exposes an existing internal testing parameter to keep the intermediate parquet shard files.

In addition, creates new option to keep the intermediate resume logging files.

New Feature Checklist

• [x] I have added or updated the docstrings associated with my feature using the NumPy docstring format
• [ ] I have updated the tutorial to highlight my new feature (if appropriate)
• [x] I have added unit/End-to-End (E2E) test cases to cover my new feature
• [ ] My change includes a breaking change
  • [ ] My change includes backwards compatibility and deprecation warnings (if possible)

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 99.53%. Comparing base (3f7a423) to head (7f6bb55).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #311 +/- ## ======================================= Coverage 99.52% 99.53% ======================================= Files 25 25 Lines 1274 1282 +8 ======================================= + Hits 1268 1276 +8 Misses 6 6 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.