sophiamaedler
commented
2 years ago Segmentation is also recalculated if resolve_sharding did not complete even though the segmentation of each individual shard completed successfully. It seems to only check for the presence of intermediate maps like below and not for the presence of a segmentation.h5 file.
[27/02/2022 21:12:21] No existing map 0 normalized found at path /fs/pool/pool-mann-viperssd/datasets/Autophagy_Screen_2/Training_Data_final/T_2.1_torin_wt/segmentation/tiles/0/0_normalized_map.npy, new one will be created
[27/02/2022 21:12:21] No existing map 1 median found at path /fs/pool/pool-mann-viperssd/datasets/Autophagy_Screen_2/Training_Data_final/T_2.1_torin_wt/segmentation/tiles/0/1_median_map.npy, new one will be created
[27/02/2022 21:12:21] No existing map 2 nucleus_segmentation found at path /fs/pool/pool-mann-viperssd/datasets/Autophagy_Screen_2/Training_Data_final/T_2.1_torin_wt/segmentation/tiles/0/2_nucleus_segmentation_map.npy, new one will be created
[27/02/2022 21:12:21] No existing map 3 nucleus_mask found at path /fs/pool/pool-mann-viperssd/datasets/Autophagy_Screen_2/Training_Data_final/T_2.1_torin_wt/segmentation/tiles/0/3_nucleus_mask_map.npy, new one will be created
[27/02/2022 21:12:21] No existing map 4 wga_mask found at path /fs/pool/pool-mann-viperssd/datasets/Autophagy_Screen_2/Training_Data_final/T_2.1_torin_wt/segmentation/tiles/0/4_wga_mask_map.npy, new one will be created
[27/02/2022 21:12:21] No existing map 5 wga_potential found at path /fs/pool/pool-mann-viperssd/datasets/Autophagy_Screen_2/Training_Data_final/T_2.1_torin_wt/segmentation/tiles/0/5_wga_potential_map.npy, new one will be created
[27/02/2022 21:12:21] No existing map 6 travel_time found at path /fs/pool/pool-mann-viperssd/datasets/Autophagy_Screen_2/Training_Data_final/T_2.1_torin_wt/segmentation/tiles/0/6_travel_time_map.npy, new one will be created
[27/02/2022 21:12:21] No existing map 7 watershed found at path /fs/pool/pool-mann-viperssd/datasets/Autophagy_Screen_2/Training_Data_final/T_2.1_torin_wt/segmentation/tiles/0/7_watershed_map.npy, new one will be created
[27/02/2022 21:12:21] Segmentation started shard 0, starting from checkpoint 0
[27/02/2022 21:12:21] Started with normalized map
[27/02/2022 21:12:31] Normalized map created
[27/02/2022 21:12:31] Started with median map
[27/02/2022 21:17:51] Median map created
[27/02/2022 21:17:51] Started with nucleus segmentation map
[27/02/2022 21:23:11] Nucleus segmentation map created
[27/02/2022 21:23:11] Started with nucleus mask map
[27/02/2022 21:23:11] Nucleus mask map created with 20538 elements
[27/02/2022 21:23:26] Filtered out due to contact limit: 190
[27/02/2022 21:23:32] Filtered out due to size limit: 96
[27/02/2022 21:23:32] Filtered out: 286
[27/02/2022 21:23:32] Started with WGA mask map
[27/02/2022 21:23:35] No treshold for cytosol segmentation defined, global otsu will be used.
[27/02/2022 21:23:51] WGA mask map created
[27/02/2022 21:23:51] Started with WGA potential map
[27/02/2022 21:23:56] WGA mask potential created
[27/02/2022 21:23:56] Started with fast marching
[27/02/2022 21:28:31] Fast marching finished
[27/02/2022 21:28:31] Started with watershed
[27/02/2022 21:42:42] Cells filtered out due to cytosol size limit: 409
[27/02/2022 21:42:42] Filtered out: 659
[27/02/2022 21:42:42] Remaining: 19880
[27/02/2022 21:42:42] watershed finished
[27/02/2022 21:42:47] saving segmentation
[27/02/2022 21:42:56] === finished segmentation ===
[28/02/2022 10:43:14] No existing map 0 normalized found at path /fs/pool/pool-mann-viperssd/datasets/Autophagy_Screen_2/Training_Data_final/T_2.1_torin_wt/segmentation/tiles/0/0_normalized_map.npy, new one will be created
[28/02/2022 10:43:14] No existing map 1 median found at path /fs/pool/pool-mann-viperssd/datasets/Autophagy_Screen_2/Training_Data_final/T_2.1_torin_wt/segmentation/tiles/0/1_median_map.npy, new one will be created
[28/02/2022 10:43:14] No existing map 2 nucleus_segmentation found at path /fs/pool/pool-mann-viperssd/datasets/Autophagy_Screen_2/Training_Data_final/T_2.1_torin_wt/segmentation/tiles/0/2_nucleus_segmentation_map.npy, new one will be created
[28/02/2022 10:43:14] No existing map 3 nucleus_mask found at path /fs/pool/pool-mann-viperssd/datasets/Autophagy_Screen_2/Training_Data_final/T_2.1_torin_wt/segmentation/tiles/0/3_nucleus_mask_map.npy, new one will be created
[28/02/2022 10:43:14] No existing map 4 wga_mask found at path /fs/pool/pool-mann-viperssd/datasets/Autophagy_Screen_2/Training_Data_final/T_2.1_torin_wt/segmentation/tiles/0/4_wga_mask_map.npy, new one will be created
[28/02/2022 10:43:14] No existing map 5 wga_potential found at path /fs/pool/pool-mann-viperssd/datasets/Autophagy_Screen_2/Training_Data_final/T_2.1_torin_wt/segmentation/tiles/0/5_wga_potential_map.npy, new one will be created
[28/02/2022 10:43:14] No existing map 6 travel_time found at path /fs/pool/pool-mann-viperssd/datasets/Autophagy_Screen_2/Training_Data_final/T_2.1_torin_wt/segmentation/tiles/0/6_travel_time_map.npy, new one will be created
[28/02/2022 10:43:14] No existing map 7 watershed found at path /fs/pool/pool-mann-viperssd/datasets/Autophagy_Screen_2/Training_Data_final/T_2.1_torin_wt/segmentation/tiles/0/7_watershed_map.npy, new one will be created
[28/02/2022 10:43:14] Segmentation started shard 0, starting from checkpoint 0
[28/02/2022 10:43:14] Started with normalized map
[28/02/2022 10:43:26] Normalized map created
[28/02/2022 10:43:26] Started with median map
[28/02/2022 10:48:51] Median map created
[28/02/2022 10:48:51] Started with nucleus segmentation map
[28/02/2022 10:54:17] Nucleus segmentation map created
[28/02/2022 10:54:17] Started with nucleus mask map
[28/02/2022 10:54:18] Nucleus mask map created with 20538 elements
[28/02/2022 10:54:32] Filtered out due to contact limit: 190
[28/02/2022 10:54:39] Filtered out due to size limit: 96
[28/02/2022 10:54:39] Filtered out: 286
[28/02/2022 10:54:39] Started with WGA mask map
[28/02/2022 10:54:41] No treshold for cytosol segmentation defined, global otsu will be used.
[28/02/2022 10:54:59] WGA mask map created
[28/02/2022 10:54:59] Started with WGA potential map
[28/02/2022 10:55:04] WGA mask potential created
[28/02/2022 10:55:04] Started with fast marching
If a segmentation has been previously successfully calculated with intermediate_output = False but the workflow needs to be reprocessed (e.g. because one shard did not segment properly) and in this reprocessing intermediate_output = True and overwrite = False all tiles are still reprocessed and not only the one that is missing the final segmentation file. It could be beneficial to introduce a check where it looks for the completed segmentation file instead of searching for the individual intermediate outputs. Only if it doesn't find the completed segmentation file does it check for intermediate outputs and try to continue where it left of.
In addition we should add some options to force reprocessing of the pipeline after a specific step. E.g. if the filtering parameters to select cells change so that not the entire pipeline needs to be rerun but only those specific steps and all downstream ones.