Very Different output from github code and online API on same image to detect nuceli

[OpenCv30]

Hi, Thank you for sharing code for the opensource community. I am trying to modify your source code as per my need but I am seeing very weird behavior. I am trying to detect nuclei boundary using - https://www.nucleaizer.org/ and GitHub source code- start_prediction_full.bat for the same image.

I have few queries in this context. Please answer them-

• Why there is so much different in output? Are you using the different trained model in online and GitHub file ( downloaded from google drive as suggested on the website)

• what should be image correct image size to run the script? I was running online API using 3000x3000 / 1500x1500 images and all of them were failing. But, now I am trying with 512x512 and it is working.

[spreka]

@OpenCv30 Thank you for taking time to experiment with our method.

Regarding the different outputs: The online tool (nucleAIzer) and the full prediction pipeline were not intended to create the same output for several reasons:

• Full prediction aims for the highest accuracy at the expense of processing time and resources. It calculates a scaling parameter (which is similar but more complicated than the nucleus size you can set in the online tool for simplicity and speed), has post-processing including multiple predictions and parameter optimization. It is fully automatic, no parameter setting is required.
• nucleAIzer offers a fast and easy way to try our model on your own data. The size you can set manually (1 parameter) substitutes a rather costly (in time and computation) step in the full pipeline. No post-processing or optimization is done.

Even though you can use the same model in both (final model in the code, general model online), you will have different results comparing to full prediction. However, you can also try fast prediction. You can change the model path to the final model if you wish - note that in this case a scale file is expected, see segmentation_params_2x.json for how to set it and \biomagdsb\matlab_scripts\cellSizeEstimateForPrediction\cellSizeDataGenerate.m should give you an idea on how the file should be created.

The online API should accept images of such sizes (though 3000x3000 is quite large) given they remain under ~20 MB unless they are to be scaled up to several folds. What did you set the nucleus size parameter to when it failed? Did you succeed in uploading them (i.e. processing started)? If so, did you try with a single image at a time? Can you please send me a sample image that failed?

[OpenCv30]

@spreka Thank you for your detailed reply. I have gone through the code and still exploring. But I like to first get clarity on the inputs-

• What is the input image as per algorithm needs in term of the number of image channel and data type-8bit/16bit data? ( online = NucleAIzer and offline = Github code)

I have tried with 1 channel - HxW image with 16-bit data- It failed ( both online/offline version) Tried with 1-channel - 8bit data -- online worked but offline failed as Unet needs 3 channels as code looks like so. Tried with 3 channel - 8 bit - both online and offline worked. My workaround as I have one channel data. RUn with one channel but make it 3-channel to run UNET by repeating the same channel 3 times.

• Now, I like to understand the flow-

As per my understanding, in full evaluation mode- algorithm first runs - MaskRCNN to get an estimate of cell size and then get resize factor to get an approximate size of nuclei 40pixel.
Now resize original image to 2x. Then call - 2 times -MaskRcNN with 2 different configurations- 2x-json and 4x-json- and estimate cell size. Also, call 7 UNET model on the original input image ( without resize) and get outputs and average it. In last do postprocessing with RCNN2x.RCNN4x,UNEt-avg outputs to get final results.

I am hoping my understanding is correct if not then plz correct me.

• Job of the first maskRCNN is to get only estimates of cell size right? Actually, in my case, first masked RCNN gives me just 25 nuclei while the final post-processing has 300+ nuclei count. How come first maskRCNN is behaving so badly.

• What is the input image size to masked rcnn-2x,maskRcnn-4x- looks like 2X of the original image is fed to both? Why you are doing this? I was expecting a pyramid of images and then run Mask-rcnn on all and then upsample and add.

• You have suggested for fater mode to run. I am seeing one .bat file to run fast. But it runs the only presegment.json -for cell size estimation. The code-\biomagdsb\matlab_scripts\cellSizeEstimateForPrediction\cellSizeDataGenerate.m - reads the maskImg and gives median/std of cell size. These are used in resizing the image. So you are suggested to run like this-

• Option-1- if estimate of cell size is not known

   {masked rcnn: get cell size} ---> <masked-rcnn-2x> and <masked-rcnn-4x> -> combine these outputs. Ignore Unet prob maps.

• Option-2- if estimate of cell size is known

  {get cell size} ---> <masked-rcnn-2x> and <masked-rcnn-4x> -> combine these outputs. Ignore Unet prob maps

Still, working on your code. I will add the next questions based on your reply. I wish I would have shared you the images. But this is a medical image so non-sharable. Regards, -sk

[spreka]

@OpenCv30

What is the input image as per algorithm needs in term of the number of image channel and data type-8bit/16bit data?

Please see the README or this issue or the documentation. I just tested the online version works with 16-bit single channel images. I understand the restrictions of handling medical data, however, without further information I cannot help you with data-related issues. Judging from your first comment

3000x3000 / 1500x1500 images and all of them were failing. But, now I am trying with 512x512 and it is working

the problem might be the large image size and/or potentially small objects on the image. To support this explanation (cannot confirm without an exact sample image) please provide your nucleus size parameter and if the image uploading finished. If these tests were performed on the same data (except e.g. cropped to create smaller images), the data appears to be correct.

About the full prediction method:

I am hoping my understanding is correct

Yes it is.

Job of the first maskRCNN is to get only estimates of cell size right? Actually, in my case, first masked RCNN gives me just 25 nuclei while the final post-processing has 300+ nuclei count. How come first maskRCNN is behaving so badly.

The first Mask R-CNN prediction is used to estimate nucleus size and morphological properties. It is intended to be confident about those objects it segments so these features can be estimated/measured on accurate data. Please see the parameter file presegment.json for the confidence value and set it to your preference: the lower the value the more objects detected.

What is the input image size to masked rcnn-2x,maskRcnn-4x- looks like 2X of the original image is fed to both? Why you are doing this?

Indeed, we scale input images to 2x. You may consider this as an initial step for our post-processing scheme where objects are expected to be either detected or segmented differently in the 2x/4x prediction. We merge these objects as defined by the optimized parameters.

Regarding fast prediction I suggested you either prepare a scale file according to cellSizeDataGenerate.m and run the final model (if you want similar results to the online version) with this additional parameter in presegment.json. Note the confidence parameter in the json files. If you do not wish to create such a file for yourself, you can rely on the full prediction pipeline to do it for you. You can halt the execution after the scale file is created, then run a fast prediction using the scale file and the final model. The full prediction pipeline will do all of these automatically.

[OpenCv30]

@spreka- I am going through every module to understand your code better. I like to ask a few queries as follows-

• In cellSizeDataGenerate.m, resizeTo = (40/median_cell_Eqv_dims_in_found_cell(using first maskRCNN with high thd =0.7)(sx+sy)/2correctionFactor,where correctionFactor is 1,2,4. Basically you are ensuring that cell size should be minimum 40pixel. Then this rezieTo dims is adjusted so that image should be divisible by 64 ( i.e 6 times by 2).

• You are using 5- stages -Resnet so size is getting divided by 2 every time. Hence good to have size multiple of two. Am I right?

• Is there any limit on how big cell size can be?

• The online version asks for a minimum cell size. Are you using it in the above equation as median_cell_Eqv_dims_in_found_cell(using first maskRCNN with high thd 0.7) matlab code- @30
  This will help you to get rid of first RCNN call on the original image. Am I right?

• To run 2x.json/4x.json, the algorithm is again resizing the image in -model.mold_inputs-based on- min_dim=self.config.IMAGE_MIN_DIM, min_scale=self.config.IMAGE_MIN_SCALE, max_dim=self.config.IMAGE_MAX_DIM, mode=self.config.IMAGE_RESIZE_MODE There are set during RCNN model building? So why now you seperately create 2x image nad 4x image and use it later. Why two step process is there-first resize image in matlab by 2x and then again modify this 2x to 1920 and 3840 ( for 512x512 image).

• In cellSizeDataGenerate.m, the unet.csv is getting overwritten every time for different correctionFactor and finally, it saves for biggest correction factor. Is it intentional or a bug ? if intentional then why?

• 2 ..even though I have done- pip install opencv-python==3.4.2.17, I am getting the same issue

  in {if pCavityFilling: for i in range(count): temp = cv2.bitwisenot(masks[:, :, i]) temp, , _ = cv2.findContours(temp, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE) masks[:, :, i] = cv2.bitwise_not(temp) '''for i in range(count): masks[:, :, i] = scipy.ndimage.binary_fill_holes(masks[:, :, i])''' } what is your temp? Can i do like this?

  temp = np.zeros_like(image) for cnt in foundCountors: drawnCnt = cv2.drawContours(cnt) temp = np.logiocal_or(temp,drawnCnt )

[spreka]

@OpenCv30

In cellSizeDataGenerate.m [...] Basically you are ensuring that cell size should be minimum 40pixel.

Exact 40 pixels to have uniformly sized objects.

Then this rezieTo dims is adjusted so that image should be divisible by 64

Yes, for the network architecture.

[...] Hence good to have size multiple of two. Am I right?

If you mean the full pipeline, this doesn't matter as we scale the images in the scripts. For fast prediction (no scaling) the prediction script handles it.

Is there any limit on how big cell size can be?

We tested on circles ranging from 1-125 pixels in diameter and found that the presegmenter model segments objects in range [7-125] accurately while the final model (trained on fix object sizes) in [7-100]. Note we expect objects to be of approximately 40 pixels size in diameter, hence the scaling.

The online version asks for a minimum cell size. Are you using it in the above equation as median_cell_Eqv_dims_in_found_cell(using first maskRCNN with high thd 0.7) matlab code- @30 This will help you to get rid of first RCNN call on the original image. Am I right?

NucleAIzer asks for an approximate (say average) size of nuclei, not minimum. Yes, it is used in a similar equation to calculate scaling and indeed substitute the first Mask R-CNN call (to save execution time).

[image sizes] are set during RCNN model building? So why now you seperately create 2x image nad 4x image and use it later.

We assembled the full prediction pipeline to achieve high accuracy on heterogeneous data (i.e. random images of different domains). We predict on 2x and 4x scaled images to detect a wider range of objects more precisely, which are then combined/merged in the post-processing step. Those Mask R-CNN parameters are set for training, then for prediction (to handle the randomly sized test images) the model gets rebuilt.

In cellSizeDataGenerate.m, the unet.csv is getting overwritten

I think we omitted this file from the pipeline, you can ignore it.

2 ..even though I have done- pip install opencv-python==3.4.2.17, I am getting the same issue

Did you run pip uninstall opencv-python before running the install command? Could you please verify that only the required version is installed in your env?

what is your temp?

temp = cv2.bitwise_not(masks[:, :, i]) means the ith object mask is inverted The referenced line is only needed if you want to fill holes in detected objects. You are free to do it any way you prefer (e.g. see this line).

[OpenCv30]

@spreka - Thank you for your reply. I am planning to write a fast prediction code in python ( no MatLab) and then full pipeline in python but looks like your post-processing is complex and a number of Matlab files has to be converted into python. So, I am focussing on the first. Also, you have suggested in the past how to do fast processing. I am doing like this but not getting as expected in the online version.

• Image-A -512x512- ran online - gave 418 nuclei for nuceli pixSize setting of -20pix
• Image-A -512x512- ran online - gave 77 nuclei for nuceli pixSize setting of -40pix
• Image-A -512x512- ran github Code- gave 324 nuclei for nuceli pixSize setting -automatic so very much dependent of the setting.

What I am doing with your Github Code- First, ran the full pipeline code and it gave size file/dimension as shown in the figure with caption-cellSizeEstimator.Updated the presegment.json as shown in the figure and ran -start_prediction_fast.bat

• original presegment - ran without scale file with threshold reduced to 0.7. The number of nuclei detected is just 25
• original image(512x512) + Modified-1 presegment - ran with scales-rcnn-1x.csv and threshold reduce to 0.6. The number of nuclie detected is just 68
• original image(512x512) +Modified-2 presegment - ran with scales-rcnn-2x.csv and threshold reduce to 0.6. The number of nuclie detected is just 386
• original image(512x512) +Modified-3 presegment - ran with scales-rcnn-4x.csv and threshold reduce to 0.6. The number of nuclie detected is just 572

Am I doing right as per your suggestion? if yes, then why there is still so much difference b/w the online output and GitHub output. Which presegmnet.json should I use for fast processing

Since, the full pipeline runs 7-UNETS + 2 MaskRCNN-2x/4x and then post-process, which is too much time-consuming. I am thinking of just do post-process with MASK_RCNN-2x and 4x which might be better than just using 2x output but inferior to the full pipeline. What is your comment on this?

[OpenCv30]

@spreka - I have done one more experiment- I know online version uses nucleiSize=20pix so in full pipeline code, I have hardcoded cell-median to 20 as shown in code.

function [median_size, std_size] = estimateCellSizeFromMask(maskImg) props = regionprops(maskImg, 'EquivDiameter'); allSize = [props.EquivDiameter];
median_size = 20;%median(allSize); std_size = std(allSize);

With this, I have generated scales-rcnn-1x/2x/4x.csv with above setting. Then ran modified1-presegment.json and modified2-presegment.json, results are as follows-

• modified1-presegment.json, gave the total number of nuclei-78 which is very close to running online version with nuclieSize-40x

• modified2-presegment.json, gave the total number of nuclei-418 which is same as running online version with nuclieSize-20x

Looks like, from my analysis, you are using-modified2-presegment.json- for any input image in fast segmentation mode. Please correct me if I am wrong.

[spreka]

@OpenCv30 We are preparing the images similarly in the online version to what you did. The differences you experience in the number of detected objects arise mainly from the scale files. The pre-segmentation model is not used for final detections, neither is the 1x scale file. 2x scale file should bring you close to the online results except that the calculated scaling should follow your data more accurately if the object sizes vary in a certain range (say larger than +/- 3 pixels). However, if they are approximately the same size, a single estimated size parameter (as in the online version) can provide sufficient detections. If you are satisfied with the fast prediction pipeline accuracy, feel free to adjust it to your needs instead of using full prediction.

[OpenCv30]

@spreka - Thank you so much for confirming and a very prompt reply :) I am going to play with fast segmentation only but if this doesn't fulfill my requiremnet then I have to go for the full pipeline. I guess you have overlooked one of my query regarding full pipeline-

Since, the full pipeline runs 7-UNETS + 2 MaskRCNN-2x/4x and then post-process, which is too much time-consuming. I am thinking of just do post-process with MASK_RCNN-2x and 4x which might be better than just using 2x output but inferior to the full pipeline. What is your comment on this?

Also, your model is trained with just Kaggle Nuclei data- Bowl 2018 or some other data too? Can you point me such dataset if it is free?

[spreka]

@OpenCv30

Since, the full pipeline runs 7-UNETS + 2 MaskRCNN-2x/4x and then post-process, which is too much time-consuming. I am thinking of just do post-process with MASK_RCNN-2x and 4x which might be better than just using 2x output but inferior to the full pipeline. What is your comment on this?

Ideed, the full pipeline takes time to execute. However, it will provide higher accuracy compared to fast (or similar) prediction. I believe you are right about the performance estimation without post-processing, I cannot claim what the trade-off will be without trying, and of course, it depends on your specific data. You can also try playing around with the size parameter as in the online version.

Also, your model is trained with just Kaggle Nuclei data- Bowl 2018 or some other data too? Can you point me such dataset if it is free?

For training we used the official DSB2018 data, further external data shared by the participants on the official thread and some from our own lab. Those that are open source have corresponding publications you can cite.

[OpenCv30]

Hi, Thank you for constantly answering my doubts 👍 I have written a python code which calls your modules similar to online version as explained in the flow diagram. This is based on our previous discussion.

But, I am seeing difference b/w number of segmented nuclei count between online and my code version. I use same input image and same nuclei size parameter (pink) in online and my code and ran code to get segmented nuclei. The following table gives the counts of nuclei found in online version ( orange) and my code version(green).

My quries-

• why there is so much difference in the nuclei count even though i am using your code as it is?

• Is it possible that online version is using different trained model weight from the freely available trained model weight which i am using?

• Is this much difference coming due to system difference- GPU/CPU etc?

Please help me to understand the differences?

[spreka]

@OpenCv30 I think you have already come to the conclusion (in this comment) that you can get very close to the performance of the online tool locally by properly setting the parameters. The differences you experience compared to your python code should (theoretically) be marginal I believe if the code follows our implementation faithfully. Even though there are obvious differences between python and matlab, you should be able to replicate an algorithm fairly well in the other. Of course, as I have not seen your implementation, it is up to you to make it follow ours. Therefore, I cannot possibly answer your question as to why those differences happen.

Is it possible that online version is using different trained model weight from the freely available trained model weight which i am using?

No, we use the same model: final model in GitHub code and first (general) model online.

Is this much difference coming due to system difference- GPU/CPU etc?

Again, as you have already got similar results locally, your hardware setup should be fine for the task. If there was a memory issue for instance, you would get an out-of-memory error; this could happen for large images.

I appreciate your commitment to analyse our code so thoroughly, but unfortunately we don't have the resources to investigate why your code gives different results than ours. The code is open source, you may use it freely to compare step-by-step where your results and ours start to be different.