Segmentation outline with Mesmer

[marcovarrone]

Hi,

I am using Mesmer to segment cell nuclei of multiplexed immunofluorescence images. To check if the segmentation works well, is it possible to obtain the segmentation outlines overlaid on the channels as described in the documentation of s3seg?

As far as I understood s3segmenter works with probability maps, so it's not possible to run it on the output of Mesmer which is masks directly. However, is there a way to run the mask-to-outline part of s3seg only?

Thank you very much

[clarenceyapp]

Hi @marcovarrone S3seg has a relatively new parameter to accept pre-computed segmentation masks and make outline files. In your .yml file under the options section, can you add

options:
 s3seg:  --nucleiRegion bypass --logSigma 15 300

to the list of s3segmenter parameters? The logSigma parameter allows you to filter out cells that deviate outside of that cell diameter range (in pixels). Feel free to leave that out if you want to accept all cell objects.

[ArtemSokolov]

Minor adjustment. Whether or not the output of a segmentation module goes through s3seg is specified here: https://mcmicro.org/parameters/specs.html#watershed

I would recommend trying the following:

options:
  s3seg: --logSigma 15 300
modules:
  segmentation:
    -
      name: mesmer
      watershed: bypass 

MCMICRO will then pass the output of Mesmer to S3Seg, automatically providing --nucleiRegion bypass, so it doesn't need to provided explicitly. As Clarence mentioned, --logSigma is optional, but it may help filter nuclear regions by size.

[nancyliy]

Riding on the request here I have the same need to create segmentation outlines from Mesmer output except I generated the mask using at the whole-cell level instead of nuclear level. What should the settings be to outline the whole-cell boundaries?

[ArtemSokolov]

Hi @nancyliy,

Clarence can correct me if I'm wrong, but when using the bypass feature of S3Seg, it actually doesn't matter what mask you pass to it. It will generate outlines for any input.

I just tested exemplar-001 with the following params.yml:

workflow:
  segmentation: mesmer
options:
  s3seg: --logSigma 15 300
modules:
  segmentation:
    -
      name: mesmer
      watershed: bypass 

where

1. It uses the default Mesmer parameters (which are supposed to generate a cell mask)
2. Mesmer's mask is correctly passed to s3seg with the bypass flag and the --logSigma parameters. (As can be found in the .command.sh file in the work/ directory associated with s3seg.)
3. There is a cellOutlines.ome.tif and nucleiOutlines.ome.tif in exemplar-001/qc/s3seg/mesmer-exemplar-001. Both images look identical, which makes sense, since s3seg doesn't know what type of mask it received.

[clarenceyapp]

Hi @nancyliy The above is right. S3seg creates outlines on the primary object (in your case, whole cells). If you pass it a mask of nuclei, then the outlines will be of the nuclei.

If you have a nuclei AND cell mask from Mesmer and you want outlines for both, then the easiest thing to do right now is run MCMICRO twice (first, pass the nuclei mask into s3seg and save the nucleiOutlines file somewhere to avoid it getting overwritten. Then rerun s3seg and pass the cell mask instead and take the cellOutlines file). As Artem mentioned, nucleiOutlines and cellOutlines have identical contents within each MCMICRO run in your case.

[nancyliy]

Thanks both for your kind and helpful input. I performed a run using mesmer segmentation method followed by s3seg outline but encountered the following error. When I used the exact same input image in mcmicro pipeline with default unmicst segmentation method, the pipeline run without error but the s3seg cell outlines showed the segmentation outcome is unsatisfactory with unmicst segmentation. Goal is to get the mcmicro pipeline to interface well with mesmer output so I can inspect the segmentation outcome.

Nextflow_Pilot2.txt

This error seems to be similar to the one I raised in issue #475. I would really appreciate if I could receive some advice on how to resolve it. Thank you.

[clarenceyapp]

@nancyliy Let me check the code today to see if there is a bug in the bypass feature. It is relatively new and I had implemented it for a specific project that only had square images. The length and the width dimensions might have gotten swapped in the code and this would not have generated an error with squares images.

For unmicst, this is really meant for segmenting nuclei. If you are expecting whole cell segmentation, then unmicst would not work. The pixel size must also be right and we have seen that all segmentation methods do not give optimal results when this does not match.

[marcovarrone]

@nancyliy which compartment are you segmenting with Mesmer? Nucleus, cytoplasm, or both?

I remember getting a similar error while using --compartment both, because the output of the segmentation was a .tif file with two channels, one for each mask, and this would not be compatible with s3seg.

[nancyliy]

@marcovarrone in my command I was trying to use --compartment both, and output of the segmentation indeed is a .tif file with the dimension (11520, 19392, 2) with 2 channels. Maybe this is why it is not compatible with s3seg and mcquant. In your experience when you set --compartment to be either nuclear or whole-cell were you able to successfully generate segmentation outline with s3seg and get quantification results Mcquant? I will be happy to try running mesmer on one compartment at a time and hopefully resolve this dimension issue.

@marcovarrone regarding input for mcmicro, my initial CODEX experiment data format is one folder with each channel as one .tiff file within the folder. I used bfconvert from bftools to convert the tiffs into one multi-channel ome.tiff image as input for mesmer (I skipped the registration workflow in mcmicro and went straight into the segmentation workflow). I wonder how did you supply the input from CODEX to mcmicro for segmentation? Thanks!

[clarenceyapp]

@marcovarrone @nancyliy there would indeed be an error from s3seg if the mask is two or more channels instead of one. We always run it on one compartment per mask. Please try one and see if it is better. Thanks!

[marcovarrone]

@nancyliy Yes, I managed to obtain nuclear and whole-cell quantification results once I ran every compartment separately.

One aspect you have to be careful of is that, regardless of whether you are obtaining the masks for nuclear and whole-cell separately or with —compartment both, the ids of the masks do not match across compartments.

I am right now working on a script to match nuclear masks with the whole-cell ones based on their overlap. If it's useful in this context I would be happy to share a link to it here once it's ready!

[nancyliy]

Thanks so much @marcovarrone and @clarenceyapp on your help! the mcmicro pipeline and mask outline indeed ran smoothly after I run it on one compartment per mask:):) @marcovarrone definitely keep us posted on how the matching of both masks go -- it's great to be learning in this together!

[clarenceyapp]

@nancyliy @marcovarrone great to hear it is now working! One easy thing to do could be to split the two channel mask into individual images and re-save them as separate files. This can be done in imageJ or programmatically. Alternatively, s3seg can compute cytoplasm rings around the nuclei to approximate whole cells (alot of papers have used this method). Just pass the nuclei mask into S3seg as before but also turn on '--segmentCytoplasm segmentCytoplasm'. The IDs will match.

[marcovarrone]

@clarenceyapp thank you for the suggestion on computing the cytoplasm rings using s3seg! I have been experimenting with it but I noticed some problems.

If I use the parameter watershed: bypass for mesmer, I noticed that the resulting segmentation masks are different than the ones returned without the parameter. Other than being .ome.tif instead of .tif as it's usually produced by Mesmer (not a problem), the masks have a lower quality, meaning that they fail to detect some nuclei.

However, in the version without watershed: bypass, those nuclei are well segmented.

I tried to see if it was a matter of nucleus size, so I put a big range for the size filter, i.e. --logSigma 1 500 but the nuclei are still not detected. I also used the ring method with --segmentCytoplasm segmentCytoplasm --cytoMethod ring so it shouldn't depend on any cytoplasm marker in the image, but again those nuclei are not detected.

I report here the full params.yml file:

``` workflow: start-at: segmentation stop-at: segmentation segmentation: mesmer options: mesmer: --image-mpp 0.5 --compartment "nuclear" --nuclear-channel 0 --batch-size 8 s3seg: --logSigma 1 500 --segmentCytoplasm segmentCytoplasm --cytoMethod ring --CytoMaskChan 7 31 35 modules: segmentation: - name: mesmer version: 0.4.0-gpu watershed: bypass ```

[clarenceyapp]

Hi @marcovarrone , can you email me your Mesmer and s3seg outputs and raw images either as a Dropbox link or attachment to Clarence AT hms.harvard.edu ? I'll take a look today to see what the issue is. You can also test my original set of parameters at top of thread in the meantime if you like. Thanks!

[marcovarrone]

@clarenceyapp I am using human patient data, so unfortunately at the moment I cannot send you the raw images. I'll try to find public or non-sensitive images for which the same problem appears.

EDIT: I also tried to run MCMICRO with watershed: yes and s3seg: --nucleiRegion bypass but I get the same result.

[clarenceyapp]

@marcovarrone ok is it alright to just send the Mesmer mask and s3seg outputs without raw channels?

[marcovarrone]

@clarenceyapp I sent you the raw DAPI image and the masks of a non-sensitive dataset in which the same thing happens. Thank you very much in advance for your help!

[clarenceyapp]

Hi @marcovarrone, thanks for your patience. I see the issue and have identified what might be causing the problem. The --bypass parameter takes in the mask image and removes cells that are outside of the --logSigma range and then regenerates a new label image using scikit-image label to update the cell IDs to be consecutive. This normally works fine for most masks that we have come across when we enforce a 1 pixel line that separates each object. However, the Mesmer mask has a label mask without this 1 pixel line and so cell objects are touching. This creates 2 problems: 1) creating outlines around cells using the existing method does not work since there is no gap (but maybe an edge detector could work. I think you saw this in the preview file where large groups of cells are being outlined as one instead of individual cells, 2) relabelling of cells after filtering out objects also does not work because relabelling relies on having disconnected objects. This is the reason UnMicst and s3segmenter impose a line between the cells.

I believe the immediate short term solution for the 2nd issue is to stop right after s3segmenter has finished, manually put the Mesmer mask in the segmentation folder and rename it to cell.ome.tif or nuclei.ome.tif to trick MCMICRO into thinking that s3segmenter correctly ran. Then rerun MCMICRO starting at quantification using the Mesmer output. This assumes that you do not need s3segmenter to filter out objects.

[marcovarrone]

@clarenceyapp Thank you for the info! I would have never guessed the problem, so your help has been invaluable. If I understood your suggestion correctly though, s3seg would be used in this way only to generate the outlines for visualization. Still, I would not be able to use it to obtain the ring around cells.

However, I can create a script to add a one-pixel gap between the masks and then run s3seg+quantification. Is there a way to run s3seg in MCMICRO without rerunning the Mesmer segmentation? I think this may be necessary if I want to change the mask between Mesmer and s3seg.

Again, thank you very much for your help :)

[ArtemSokolov]

@marcovarrone You shouldn't need to move files around by hand. By default, S3seg doesn't get run on Mesmer, and Mesmer outputs get automatically copied to the segmentation/ folder. We changed this default behavior with

modules:
  segmentation:
    -
      name: mesmer
      watershed: bypass 

above, so you would just take these lines out of your params.yml, if you wanted to skip S3seg.

If you do have a way to add 1-pixel gaps and want to continue running S3seg, then you can start MCMICRO at that step without rerunning Mesmer by adding start-at: watershed to the workflow section:

workflow:
  segmentation: mesmer
  start-at: watershed

[ciszew]

@clarenceyapp Thank you for the info! I would have never guessed the problem, so your help has been invaluable. If I understood your suggestion correctly though, s3seg would be used in this way only to generate the outlines for visualization. Still, I would not be able to use it to obtain the ring around cells.

However, I can create a script to add a one-pixel gap between the masks and then run s3seg+quantification. Is there a way to run s3seg in MCMICRO without rerunning the Mesmer segmentation? I think this may be necessary if I want to change the mask between Mesmer and s3seg.

Again, thank you very much for your help :)

Hi @marcovarrone I was wondering if you were able to solve the issue with segmentation outlines created by mesmer/s3seg and if yes would you be able to share your approach? Thanks

[marcovarrone]

Hi @ciszew Unfortunately, I am not sure I have the code anymore. However, as far as I remember my approach was to run skimage.morphology.dilation followed by skimage.morphology.erosion. I am not 100% sure if I had to implement other small tricks or if it worked immediately.

Sorry for not being more helpful than this :/

[ke2rchan]

hello! @ArtemSokolov

I have just ran the script you have suggested: modules: segmentation:

  name: mesmer
  watershed: bypass 

But was returned the following error: Would appreciate any help to get around this!

[ArtemSokolov]

Hi @ke2rchan,

That looks like a parsing error. Can you confirm that you have the correct indentation in your YAML config file?

I see that you posted the following above:

modules:
segmentation:
-
name: mesmer
watershed: bypass

But it should be:

modules:
  segmentation:
    -
      name: mesmer
      watershed: bypass 

Please note that the above are spaces and not tabs.

[ke2rchan]

@ArtemSokolov Hello, thank you so so much for your help and for the prompt response! That solved my issue. However, the outline ouput is slightly different from the one generated by unmicst, where each cell is fully outlined/circled.

Could I then check with you, for the s3seg outline output, if there is any way I can quantify how many cells are identified in each outline? (since the mesmer+s3seg outline outlines several cells at once rather than each individual cell in unmicst+s3seg)(see image attached below)

For reference, the .yml I used is as follows: workflow:
start-at: segmentation stop-at: downstream
segmentation: mesmer downstream: naivestates

options: s3seg: --logSigma 3 300
mesmer: --image-mpp 0.26154 naivestates: --log no

modules:
segmentation:

• name: mesmer watershed: bypass

[ArtemSokolov]

Hi @ke2rchan,

This looks like a possible issue with the segmentation accuracy. @clarenceyapp developed s3seg and may be able to help with parameter settings.