Applicability to other imaging modalities

[SJayYang]

Hi, was wondering if I could use the model out of the box to try to perform cell segmentation using other tissue imaging modalities (fluorescent confocal images)

If not, what are some things I should look into modifying?

[FabianHoerst]

I cannot provide a general answer to that. Ideally, you should test this on a few sample images. However, this will only work for binary cell segmentation using the binary head of the model (called NP branch in the paper). I suspect that the image modalities are too diverse and the model will fail. You can see how the results are generated for the binary case in the inference script for the MoNuSeg dataset. However, I assume that fine-tuning with an annotated dataset is necessary. For that, you should create a fine-tuning script similar to the cell_segmentation/run_cellvit.py script and use it for your own dataset. You would load the pre-trained model, then load annotated fluorescent confocal images and train on them. The hyperparameters for creating the model and loading the state dict are stored as a dictionary in the provided PyTorch checkpoints.

[SJayYang]

Could you point me to which script specifically I could try for the binary segmentation task for my image modality? For example, like what you meant by the inference script for the MoNuSeg dataset. Does that inference script expect all of the extra data aside from purely the png images?

[FabianHoerst]

You'll have to write your own script as you need to write your own dataloader etc. if this are not WSI files. If you have some type of .png files, you can orientate at this script: inference_cellvit_monuseg.py If you have WSI files, then simply follow the inference script cell_detection.py, but adapt the postprocessing according to the inference_cellvit_monuseg.py script to just get the binary masks. This repo was originally not intended to be used for binary segmentation, as the main focus was to improve detection and classification, not segmentations.