Is your feature request related to a problem? Please describe.
If the Rosetta pipeline crashes in the middle of a process, there isn't a recovery scheme in place to prevent needing to re-run it in its entirety. The functions where this has been problematic:
copy_image_files
generate_rosetta_test_imgs
create_tiled_comparison
add_source_channel_to_tiled_image
compensate_image_data
Even over Ethernet connection to the NAS, certain processes can still take several hours.
Describe the solution you'd like
These functions, and the notebook itself, need to bypass files and directories that have already been created. This involves adding additional checks similar to what we do in the Pixie pipeline.
I think #403 should help to speed up the testing workflow. This would still be worth doing for the actual compensation step though, since is run across the entire cohort.
Is your feature request related to a problem? Please describe.
If the Rosetta pipeline crashes in the middle of a process, there isn't a recovery scheme in place to prevent needing to re-run it in its entirety. The functions where this has been problematic:
copy_image_files
generate_rosetta_test_imgs
create_tiled_comparison
add_source_channel_to_tiled_image
compensate_image_data
Even over Ethernet connection to the NAS, certain processes can still take several hours.
Describe the solution you'd like
These functions, and the notebook itself, need to bypass files and directories that have already been created. This involves adding additional checks similar to what we do in the Pixie pipeline.