add mask as channel in preprocessing

[Soorya19Pradeep]

Changes in preprocessing in the branch:

1. Removed flatfielding in preprocessing.
2. Dataset normalization values stored in plate zattrs file.
3. FOV statistics stored in position zattrs.
4. 3D segmentation transferd to branch.

[mattersoflight]

Thanks @Soorya19Pradeep. Looks good to me. Since @ziw-liu is working on rm-flatfield-from-train branch, please merge after his review.

One question I couldn't get an answer to while scanning the code: Does the data loader assume that the mask channel has a specific name (e.g., otsu_mask)? If yes, we need to update that part of the code to use the metadata you are writing.

We can merge this PR if the masks are now assigned the channel name that links them to the source (e.g., nuc_mask, membrane_mask). We can refine the data-loading/augmentation strategy when we meet next offline.

[Soorya19Pradeep]

Hello @mattersoflight. Thank you for looking into the PR.

The current version of the code names the mask channels based on the channel name used for segmentation, adding '_mask' at the the end of the channel name. For instance, if the channel to be segmented is 'Deconvolved-Nucl'. The mask channel name will be 'Deconvolved-Nucl_mask'. I have tested the chnages on this dataset if you want to see the the results. /local/scratch/groups/cmanalysis.grp/microDL_SP/Input/all_pos_Phase5e-4_Denconv_Nuc8e-4_Mem8e-4_pad15_bg50_registeredNGFF_32slices.zarr

[mattersoflight]

The mask channel name will be 'Deconvolved-Nucl_mask'. I have tested the changes on this dataset if you want to see the results.

👍🏼 After browsing the datasets, I realized that masks are currently written as float32 arrays. They should be written as bool and assigned metadata such that they appear as napari label layers (https://napari.org/stable/howtos/layers/labels.html).

[mattersoflight]

Another suggestion: if the 3D segmentation code is now part of the iohub_dependency branch, please delete the preprocessing_3DSegmentation branch. Looking at the commit graph, the commit history looks quite linear between other 3 active branches.

[Soorya19Pradeep]

The mask channel name will be 'Deconvolved-Nucl_mask'. I have tested the changes on this dataset if you want to see the results.

👍🏼 After browsing the datasets, I realized that masks are currently written as float32 arrays. They should be written as bool and assigned metadata such that they appear as napari label layers (https://napari.org/stable/howtos/layers/labels.html).

On discussion we decided to store the mask as the same datatype as the segmented channel. On loading the mask as label layer to napari, the features in napari related to visualization of label layers not flexible to use. So it doesn't add an advantage for visualization.

[Soorya19Pradeep]

@ziw-liu, @mattersoflight, I have added a documentation markdown file explaining the changes on dataset during preprocessing. Please let me know if I missed out something and I will add that in.