feat: Reuse spliced alignment information based on md5sum

[berntpopp]

Description: Extend the existing mechanism for reusing indices based on md5sums to also reuse spliced alignment information. By calculating and storing the md5sums of the plasmid and reference files, the pipeline can determine if the spliced alignment has already been performed for a given combination and reuse the existing output. This will save computational resources and time, especially for large datasets.

Tasks:

• Generate md5sum hashes for both the plasmid and reference input files.
• Implement logic to save and load spliced alignment results using a directory structure or filenames that incorporate the md5sum hashes.
• Modify the pipeline to check for existing spliced alignment outputs before performing the spliced alignment step.
• Update documentation to explain how spliced alignment information is reused based on md5sums.
• Add tests to ensure that spliced alignments are correctly reused when the input files are unchanged.

Benefits:

• Reduces computational time by avoiding redundant spliced alignment operations.
• Makes the pipeline more efficient, particularly when processing large or repetitive datasets.

Related Issues:

• 39

• 12

[berntpopp]

Summary of the Implementation Plan:

1. MD5 Checksum Mechanism:

   • Implement a mechanism to calculate and store MD5 checksums for files involved in the pipeline (e.g., plasmid FASTA files, indices, spliced alignments).
   • Store these checksums in a md5sum.txt file within the relevant folders to enable checking for existing results before recalculating.

2. Reuse of Intermediate Files:

   • Extend the pipeline to reuse existing intermediate files (such as indices and spliced alignments) based on the stored MD5 checksums. This helps to avoid redundant computations, improving efficiency.
   • Intermediate files should be stored in a dedicated intermediate subfolder within the plasmid input directory. This ensures they can be reused across different runs.

3. File Naming and Uniqueness:

   • To avoid conflicts, all intermediate files should be named based on the base names of the relevant input files (human genome, plasmid, sequencing files). This ensures that files are unique to the specific combination of inputs.
   • Update the pipeline to use this naming convention consistently for spliced alignments, indices, and other intermediate files.

4. Avoid Redundant MD5 Calculations:

   • Skip the MD5 checksum calculation for large files like the human reference genome to avoid performance bottlenecks, as these files are unlikely to change between runs.

5. Copying Files to Output Folder:

   • Ensure that copies of the relevant intermediate files are saved in the output folder for the current run. This makes it easier to access these files later if needed for debugging or further analysis.

Comments on the Current Code:

1. Pipeline Functionality:

   • The run_pipeline function now supports the reuse of intermediate files based on MD5 checksums. However, ensure that the checksums are stored and retrieved correctly for each run to maintain efficiency.
   • The plasmid_intermediate_folder is correctly used to store intermediate files, but care should be taken to avoid overwriting files unless explicitly requested.

2. Utils Functionality:

   • The utility functions in utils.py (such as calculate_md5, write_md5sum, load_md5sum, and check_md5sum) provide the foundation for the MD5-based reuse mechanism. Ensure these functions are well-tested and handle edge cases, such as missing or corrupted files.
   • The copy_file_to_folder function ensures that intermediate files are copied to the output directory. This is essential for debugging and downstream analyses.

3. File Naming:

   • The current implementation in run_pipeline.py uses a combination of the human reference, plasmid, and sequencing file base names to generate unique file names. This approach should work well to avoid file conflicts.

Next Steps:

1. Testing:

   • Thoroughly test the pipeline with different combinations of inputs to ensure that the MD5-based reuse mechanism is functioning correctly. Pay attention to cases where intermediate files might be incorrectly reused or overwritten.

2. Performance Optimization:

   • Profile the pipeline to identify any remaining bottlenecks, particularly in the handling of large files. Consider further optimizations if needed.

3. Documentation:

   • Update the documentation to clearly explain how the MD5-based reuse mechanism works, including any caveats or limitations. Provide examples of how users can configure and run the pipeline with the new features.

4. Feedback and Iteration:

   • Share the updated implementation with collaborators or users to gather feedback. Be prepared to make adjustments based on their experiences and suggestions.