Sparsity constraint in channel exchanging

[hljeong]

Hello, Thank you for your very interesting work! I was planning on experimenting with CEN but I couldn't seem to find the implementation of the sparsity constraint in channel exchanging, as mentioned in Section 3.3, that channel exchanging is only performed in different (disjoint) sub-parts for different modalities. Would you be able to point me to where in the model is this implemented?

Thanks.

[yikaiw]

Hi, for semantic segmentation, the sparsity constraint is implemented at: (1) different (disjoint) sub-parts, https://github.com/yikaiw/CEN/blob/158e31338165f9200dd0e2f64694463fd8a2efd0/semantic_segmentation/main.py#L396-L400 (2) adding the loss of the sparsity constraint, https://github.com/yikaiw/CEN/blob/158e31338165f9200dd0e2f64694463fd8a2efd0/semantic_segmentation/main.py#L282-L283

For image-to-image translation, the sparsity constraint is implemented at: (1) different (disjoint) sub-parts, https://github.com/yikaiw/CEN/blob/158e31338165f9200dd0e2f64694463fd8a2efd0/image2image_translation/main.py#L112-L119 (2) adding the loss of the sparsity constraint, https://github.com/yikaiw/CEN/blob/158e31338165f9200dd0e2f64694463fd8a2efd0/image2image_translation/main.py#L208-L213

[hljeong]

Thank you for your reply! However I am still confused as to where the sparsity constraint in terms of channel exchanging is implemented, as the sections of code you referenced seem to be applying the sparsity constraint to the loss calculation.

I am mainly confused about https://github.com/yikaiw/CEN/blob/158e31338165f9200dd0e2f64694463fd8a2efd0/semantic_segmentation/models/modules.py#L12-L15 which seems to exchange channels within all of x[0] and x[1], instead of disjoint sub-parts of them.

[yikaiw]

Hi, take semantic segmentation as an example: We apply the sparsity constraints on disjoint sub-parts of BN scaling factors in, https://github.com/yikaiw/CEN/blob/158e31338165f9200dd0e2f64694463fd8a2efd0/semantic_segmentation/main.py#L396-L400 In the case of two modalities, we divide channels into two disjoint sub-parts, which is implemented by adding param[:len(param) // 2 and param[len(param) // 2:] to slim_params. Followed up by the sparsity loss on slim_params, which means only the sub-parts in slim_params are constrained by L1.

We find if a channel is out of the sparsity constraints (L1), its BN scaling factor can be hardly lower than the small threshold during training. Therefore we check the criteria for channel exchanging directly on the whole channels, https://github.com/yikaiw/CEN/blob/158e31338165f9200dd0e2f64694463fd8a2efd0/semantic_segmentation/models/modules.py#L12-L15 Since constraining half (disjoint sub-parts) of the channels is already implemented in main.py, checking the exchanging criteria on the whole channels is almost equivalent to disjoint sub-parts.

[hljeong]

That makes sense. Thank you for your detailed explanation!

[rginjapan]

@yikaiw 你好 （1） 我不是特别理解用L1norm来惩罚 scale factor 在loss function 的意义，这一项在loss function里不就是让 scale factor 越来越小么 简单的来说。能不能稍微解释一下呢， 谢谢🙏 （2）这里的certain portion 就是disjoint 的那部分的意思是么？