Improve creation of polygons for small segmented objects

[scottberry]

I tried to segment the FISH spots and save as polygons but it didn’t work very well:

Copy of email correspondence with @hackermd :

I get the impression that even at maximum zoom, I am not seeing the actual segmentation layer in the viewer (the segmentations look much cleaner in the debug plots, after setting IMAGE_RESIZE_FACTOR=1).

We apply some simplification to the polygons before sending them to the client for visualisation: https://github.com/TissueMAPS/TmLibrary/blob/95e2850eeef27356d42ede293ece3d75c253dba6/tmlib/models/mapobject.py#L1028

Otherwise it would be too much data to transfer. We might want to expose the “tolerance” parameter to be able to set this for each object type differently.

Are the polygons used for the viewer only? Or are these actually used to compute the feature values?

Only for the viewer. Features are computed on the images.

Many of them are discarded during object saving, … but I guess this could also be avoided if a higher resolution is used?

Not really. This happens at maximal resolution. This has to do with the way we “polygonize” objects: https://github.com/TissueMAPS/TmLibrary/blob/95e2850eeef27356d42ede293ece3d75c253dba6/tmlib/image.py#L778

What probably creates a problem for your use case is that we apply morphological opening to the object before finding the contour: https://github.com/TissueMAPS/TmLibrary/blob/95e2850eeef27356d42ede293ece3d75c253dba6/tmlib/image.py#L821

This may also be the cause for the other issue that you reported (loss of nuclei after separate_clumps).

The “smoothing” became necessary for cases when objects have weird protrusions, but the approach could certainly be further improved.

If I omit “represent as polygons”, can I still download the segmentation images?

No. Segmentations are stored as geometries and not in form of images.

[hackermd]

To summarize: We currently apply morphological opening to segmented objects before finding their contour. This is necessary to remove thin protrusions, which would result in invalid polygons. We must ensure, however, that we don't loose any objects or significantly alter the shape of the objects, which is particularly problematic for small objects.

[hackermd]

A side note: The effect is not as dramatic as it seems based on the above screenshot, because we also simplify the polygons for visualization (they are stored at original resolution) to reduce the amount of data that needs to be transferred and rendered client side. This could be further fine tuned for particular object types, but we should track that in a separate issue.

To see the actual segmentation result (reconstructed from the stored polygons), you can download the label image via tmclient using the download_segmentation_image() method. Be careful with download_segmentation_image_file(), because in your case you have probably more than 2^16 objects per image, which would exceed the bit depth of the PNG file.