Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning (DPLNet)
🔮 Welcome to the official code repository for Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning. We're excited to share our work with you, please bear with us as we prepare the code and demo. Stay tuned for the reveal!
🔮 Our work has been accepted by IROS 2024 (Oral presentation)!
Illustration of Idea
💡 Previous multimodal methods often need to fully fine-tune the entire network, which are training-costly due to massive parameter updates in the feature extraction and fusion, and thus increases the deployment burden of multimodal semantic segmentation. In this paper, we propose a novel and simple yet effective dual-prompt learning paradigm, dubbed DPLNet, for training-efficient multimodal semantic segmentation.
Framework
Overview architecture of the proposed DPLNet, which adapts a frozen pre-trained model using two specially designed prompting learning modules, MPG for multimodal prompt generation and MFA for multimodal feature adaption, with only a few learnable parameters to achieve multimodal semantic segmentation in a training-efficient way. More details can be seen in the paper.
Implementation
Requirements
The code has been tested and verified using PyTorch 1.12.0 and CUDA 11.8. However, compatibility with other versions is also likely.
Dataset Preparation
NYUDv2 dataset can be download here NYUDv2. # change the data root in ./RGBD/configs/nyuv2.json
Pretrained Model Weights
| We provide our trained checkpoints for results reproducibility. | Dataset | url | mIoU(SS/MS) |
|---|---|---|---|
| NYUv2 | Model | 58.3/59.3 |
Put the segformer pre-trained weight. in the following file (We use segformer.b5.640x640.ade.160k.pth in our paper).
vim ./RGBD/toolbox/models/segformermodels/backbones/mix_transformer_ourprompt_proj.py # line 457Training
# run for NYUV2
cd ./RGBD
python train.pyEvaluation
# run for NYUV2,put the pretrained weight on your folder.
# example: python evaluate.py --logdir /mnt/DATA/shaohuadong/DPLNet/NYUDv2
cd ./RGBD
python evaluate.py --logdir "MODEL PATH"Experiments
🎏 DPLNet achieves state-of-the-art performance on challenging tasks, including RGB-D Semantic Segmentation, RGB-T Semantic Segmentation, RGB-T Video Semantic Segmentation, RGB-D SOD and RGB-T SOD. Note that 'SS' and 'MS' refer to single-scale and multi-scale testing, respectively. Additional results can be found in our paper.
Results on NYUDv2 (RGB-D Semantic Segmentation)
| Methods | Backbone | Total Params | Learnable Params | mIoU |
|---|---|---|---|---|
| CMX-B5 (MS) | MiT-B5 | 181.1 | 181.1 | 56.9 |
| CMXNeXt (MS) | MiT-B4 | 119.6 | 119.6 | 56.9 |
| DFormer-L (MS) | DFormer-L | 39.0 | 39.0 | 57.2 |
| DPLNet (SS) (Ours) | MiT-B5 | 88.58 | 7.15 | 58.3 |
| DPLNet (MS) (Ours) | MiT-B5 | 88.58 | 7.15 | 59.3 |
Results on SUN RGB-D (RGB-D Semantic Segmentation)
| Methods | Backbone | Total Params | Learnable Params | mIoU |
|---|---|---|---|---|
| CMX-B4 (MS) | MiT-B4 | 139.9 | 139.9 | 52.1 |
| CMX-B5 (MS) | MiT-B5 | 181.1 | 181.1 | 52.4 |
| CMXNeXt (MS) | MiT-B4 | 119.6 | 119.6 | 51.9 |
| DFormer-B (MS) | DFormer-B | 29.5 | 29.5 | 51.2 |
| DFormer-L (MS) | DFormer-L | 39.0 | 39.0 | 52.5 |
| DPLNet (SS) (Ours) | MiT-B5 | 88.58 | 7.15 | 52.1 |
| DPLNet (MS) (Ours) | MiT-B5 | 88.58 | 7.15 | 52.8 |
Results on MFNet (RGB-T Semantic Segmentation)
| Methods | Backbone | Total Params | Learnable Params | mIoU |
|---|---|---|---|---|
| EGFNet | ResNet-152 | 201.3 | 201.3 | 54.8 |
| MTANet | ResNet-152 | 121.9 | 121.9 | 56.1 |
| GEBNet | ConvNeXt-S | - | - | 56.2 |
| CMX-B2 | MiT-B2 | 66.6 | 66.6 | 58.2 |
| CMX-B4 | MiT-B4 | 139.9 | 139.9 | 59.7 |
| CMNeXt | MiT-B4 | 119.6 | 119.6 | 59.9 |
| DPLNet (Ours) | MiT-B5 | 88.58 | 7.15 | 59.3 |
Results on PST900 (RGB-T Semantic Segmentation)
| Methods | Backbone | Total Params | Learnable Params | mIoU |
|---|---|---|---|---|
| EGFNet | ResNet-152 | 201.3 | 201.3 | 78.5 |
| MTANet | ResNet-152 | 121.9 | 121.9 | 78.6 |
| GEBNet | ConvNeXt-S | - | - | 81.2 |
| EGFNet-ConvNext | ConvNeXt-B | - | - | 85.4 |
| CACFNet | ConvNeXt-B | 198.6 | 198.6 | 86.6 |
| DPLNet (Ours) | MiT-B5 | 88.58 | 7.15 | 86.7 |
Results on MVSeg (RGB-T Video Semantic Segmentation)
| Methods | Backbone | Total Params | Learnable Params | mIoU |
|---|---|---|---|---|
| EGFNet | ResNet-152 | 201.3 | 201.3 | 53.4 |
| MVNet | - | 88.4 | 88.4 | 54.5 |
| DPLNet (Ours) | MiT-B5 | 88.58 | 7.15 | 57.9 |
Acknowledgement
This repository is partially based on our previous open-source release EGFNet.
Citation
⭐ If you find this repository useful, please consider giving it a star and citing it:
@article{dong2023efficient,
title={Efficient multimodal semantic segmentation via dual-prompt learning},
author={Dong, Shaohua and Feng, Yunhe and Yang, Qing and Huang, Yan and Liu, Dongfang and Fan, Heng},
journal={arXiv preprint arXiv:2312.00360},
year={2023}
}