How to embed diffSLIC into a superpixel segmentation model?

[cbachen1997]

Dear author, hello!

Thank you for providing such excellent work!

I hope to embed diffSLIC on the basis of a backbone for training a superpixel segmentation network. Therefore, I have carefully read the paper and code of HCFormer as well as the DiffSLIC.

However, the complex code under the detectron2 framework, as well as the implementation of diffSLIC, has caused me some confusion.

I would like to ask you that how to train diffSLIC? Specifically, which part was trainable? I would like to ask if this work can be applied to Superpixel_FCN?

[perrying]

embed diffSLIC on the basis of a backbone for training a superpixel segmentation network.

There are many way to embed diffSLIC into a backbone. For example, the superpixel sampling network style (or SuperpixelFCN style) is below:

pixel_feats = neural_nets(input_image)
diffslic_fn = DiffSLIC(n_spixels=expected_number_of_superpixels, n_iter=k_means_iterations, tau=temperature_for_softmax, stable=True)
clst_feats, p2s_assign, s2p_assign = diffslic_fn(pixel_feats)
# cls_feats are fed into a next process.

The HCFormer style (i.e., the downsampled feature map is used as the superpixel seeds) is as follows:

downsampled_feature = downsampling(input_feature)
# "n_spixels" is not required when "clst_feats" is given in forward function.
diffslic_fn = DiffSLIC(n_spixels=None, n_iter=k_means_iterations, tau=temperature_for_softmax, stable=True)
clst_feats, p2s_assign, s2p_assign = diffslic_fn(input_feature, clst_feats=downsampled_feature)
# cls_feats are fed into a next layer.

In HCFormer, we incorporate the above process into every downsampling layer.

Note that p2s_assign and s2p_assign may be used to decode and encode superpixel features by spixel_upsampling and spixel_downsampling. In HCFormer, only p2s_assign is used to decode the network output.

Specifically, which part was trainable?

diffSLIC has no trainable parameters. Instead, we train neural net's weights through diffSLIC.

[cbachen1997]

It is so kind of you to provide such a detailed answer to my question so quickly. I'll follow your suggestions and give it a try. Thank you once again for your valuable input. Have a good day!