Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation. (arXiv:2401.16923v1 [cs.CV]) https://ift.tt/BrIquCv
Integrating information from multiple modalities enhances the robustness of
scene perception systems in autonomous vehicles, providing a more comprehensive
and reliable sensory framework. However, the modality incompleteness in
multi-modal segmentation remains under-explored. In this work, we establish a
task called Modality-Incomplete Scene Segmentation (MISS), which encompasses
both system-level modality absence and sensor-level modality errors. To avoid
the predominant modality reliance in multi-modal fusion, we introduce a
Missing-aware Modal Switch (MMS) strategy to proactively manage missing
modalities during training. Utilizing bit-level batch-wise sampling enhances
the model's performance in both complete and incomplete testing scenarios.
Furthermore, we introduce the Fourier Prompt Tuning (FPT) method to incorporate
representative spectral information into a limited number of learnable prompts
that maintain robustness against all MISS scenarios. Akin to fine-tuning
effects but with fewer tunable parameters (1.1%). Extensive experiments prove
the efficacy of our proposed approach, showcasing an improvement of 5.84% mIoU
over the prior state-of-the-art parameter-efficient methods in modality
missing. The source code will be publicly available at
https://ift.tt/onK6P48.
Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation. (arXiv:2401.16923v1 [cs.CV])
https://ift.tt/BrIquCv
Integrating information from multiple modalities enhances the robustness of scene perception systems in autonomous vehicles, providing a more comprehensive and reliable sensory framework. However, the modality incompleteness in multi-modal segmentation remains under-explored. In this work, we establish a task called Modality-Incomplete Scene Segmentation (MISS), which encompasses both system-level modality absence and sensor-level modality errors. To avoid the predominant modality reliance in multi-modal fusion, we introduce a Missing-aware Modal Switch (MMS) strategy to proactively manage missing modalities during training. Utilizing bit-level batch-wise sampling enhances the model's performance in both complete and incomplete testing scenarios. Furthermore, we introduce the Fourier Prompt Tuning (FPT) method to incorporate representative spectral information into a limited number of learnable prompts that maintain robustness against all MISS scenarios. Akin to fine-tuning effects but with fewer tunable parameters (1.1%). Extensive experiments prove the efficacy of our proposed approach, showcasing an improvement of 5.84% mIoU over the prior state-of-the-art parameter-efficient methods in modality missing. The source code will be publicly available at https://ift.tt/onK6P48.
via cs.RO updates on arXiv.org http://arxiv.org/