SiamSeg: Self-Training with Contrastive Learning for Unsupervised Domain Adaptation Semantic Segmentation in Remote Sensing

GitHub Issues or Pull Requests GitHub forks GitHub Repo stars

Prediction results of our proposed method.

Network Architectural and Cross-domain Image Different.

News:

[2024/11/28] ✨✨This work was submitted to IEEE TMM.
[2024/10/17] ✨✨The arxiv paper is available.
[2024/10/13] 🔥🔥Update the code and scripts.

1. Creating Virtual Environment

Install the necessary dependencies:

torch=1.10.2,torchvision=0.11.3 and mmcv-full=1.5.0.

Recommended use of conda virtual environments

conda create -n SiamSeg python==3.8 -y
conda activate SiamSeg

pip command to install torch && torchvision && mmcv-full

pip install torch==1.10.2+cu111 torchvision==0.11.3+cu111 -f https://download.pytorch.org/whl/torch_stable.html 
pip install mmcv-full==1.5.0 -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.10.0/index.html
pip install kornia matplotlib prettytable timm yapf==0.40.1

for CN user:

pip install torch==1.10.2+cu111 -f https://mirror.sjtu.edu.cn/pytorch-wheels/cu111/?mirror_intel_list
pip install torchvision==0.11.3+cu111 -f https://download.pytorch.org/whl/torch_stable.html 
pip install mmcv-full==1.5.0 -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.10.0/index.html
pip install kornia matplotlib prettytable timm yapf==0.40.1

Installation of the reference document refer:

Torch and torchvision versions relationship.

Version relationship of mmcv and torch.

2.Preparation of data sets

We selected Postsdam, Vaihingen and LoveDA as benchmark datasets and created train, val, test lists for researchers.

2.1 Download of datasets

ISPRS Potsdam

The Potsdam dataset is for urban semantic segmentation used in the 2D Semantic Labeling Contest - Potsdam.

The dataset can be requested at the challenge homepage. The '2_Ortho_RGB.zip', '3_Ortho_IRRG.zip' and '5_Labels_all_noBoundary.zip' are required.

ISPRS Vaihingen

The Vaihingen dataset is for urban semantic segmentation used in the 2D Semantic Labeling Contest - Vaihingen.

The dataset can be requested at the challenge homepage. The 'ISPRS_semantic_labeling_Vaihingen.zip' and 'ISPRS_semantic_labeling_Vaihingen_ground_truth_eroded_COMPLETE.zip' are required.

LoveDA

The data could be downloaded from Google Drive here.

Or it can be downloaded from zenodo, you should run the following command:


cd /{your_project_base_path}/SiamSeg/data/LoveDA

# Download Train.zip
wget https://zenodo.org/record/5706578/files/Train.zip
# Download Val.zip
wget https://zenodo.org/record/5706578/files/Val.zip
# Download Test.zip
wget https://zenodo.org/record/5706578/files/Test.zip

2.2 Data set preprocessing

Place the downloaded file in the corresponding path The format is as follows

SiamSeg/
├── data/
│   ├── LoveDA/
│   │   ├── Test.zip
│   │   ├── Train.zip
│   │   └── Val.zip
├── ├── Potsdam_IRRG_DA/
│   │   ├── 3_Ortho_IRRG.zip
│   │   └── 5_Labels_all_noBoundary.zip
├── ├── Potsdam_RGB_DA/
│   │   ├── 2_Ortho_RGB.zip
│   │   └── 5_Labels_all_noBoundary.zip
├── ├── Potsdam_IRRG_DA/
│   │   ├── ISPRS_semantic_labeling_Vaihingen.zip
│   │   └── ISPRS_semantic_labeling_Vaihingen_ground_truth_eroded_COMPLETE.zip

Potsdam

python tools/convert_datasets/potsdam.py data/Potsdam_IRRG/ --clip_size 512 --stride_size 512
python tools/convert_datasets/potsdam.py data/Potsdam_RGB/ --clip_size 512 --stride_size 512

Vaihingen

python tools/convert_datasets/vaihingen.py data/Vaihingen_IRRG/ --clip_size 512 --stride_size 256

LoveDA

cd data/LoveDA
unzip Train.zip, Val.zip, Test.zip

3.Training

3.1 Preparation of pre-trained models

mit_b5.pth : We provide a script mit2mmseg.py in the tools directory to convert the key of models from the official repo to MMSegmentation style.

python tools/model_converters/mit2mmseg.py ${PRETRAIN_PATH} ./pretrained

Or you can download it from google drive.

The structure of the file is as follows

SiamSeg/
├── pretrained/
│   ├── mit_b5.pth (needed)
│   └── ohter.pth  (option)

3.2 Potsdam IRRG to Vaihingen IRRG

tips

When using distributed training scripts under linux, you need to set the permissions of the training scripts due to the file permissions.
cd SiamSeg
chmod 777 ./tools/dist_train.sh
chmod 777 ./tools/dist_test.sh

# Potsdam IRRG to Vaihingen IRRG
# CUDA_VISIBLE_DEVICES Visible GPU ids are 0-3 Total four GPU processors
# PORT Sets the communication port of the master for distributed training.
# The last 4 indicates the total number of GPUs used
CUDA_VISIBLE_DEVICES=0,1,2,3  PORT=10985 \
 ./tools/dist_train.sh \
 configs/SiamSeg/siamseg_daformer_sepaspp_mitb5_512x512_40k_PotsdamIRRG_2_VaihingenIRRG.py 4

3.3 Potsdam RGB to Vaihingen IRRG

# Potsdam RGB to Vaihingen IRRG
# CUDA_VISIBLE_DEVICES Visible GPU ids are 0-3 Total four GPU processors
# PORT Sets the communication port of the master for distributed training.
# The last 4 indicates the total number of GPUs used
CUDA_VISIBLE_DEVICES=0,1,2,3  PORT=10985 \
 ./tools/dist_train.sh \
 configs/SiamSeg/siamseg_daformer_sepaspp_mitb5_512x512_40k_PotsdamRGB_2_VaihingenIRRG.py 4

3.4 Vaihingen IRRG to Potsdam IRRG

# Potsdam IRRG to Vaihingen IRRG
# CUDA_VISIBLE_DEVICES Visible GPU ids are 0-3 Total four GPU processors
# PORT Sets the communication port of the master for distributed training.
# The last 4 indicates the total number of GPUs used
CUDA_VISIBLE_DEVICES=0,1,2,3  PORT=10985 \
 ./tools/dist_train.sh \
 configs/SiamSeg/siamseg_daformer_sepaspp_mitb5_512x512_40k_VaihingenIRRG_2_PotsdamIRRG.py 4

3.5 Vaihingen IRRG to Potsdam RGB

# Potsdam IRRG to Vaihingen IRRG
# CUDA_VISIBLE_DEVICES Visible GPU ids are 0-3 Total four GPU processors
# PORT Sets the communication port of the master for distributed training.
# The last 4 indicates the total number of GPUs used
CUDA_VISIBLE_DEVICES=0,1,2,3  PORT=10985 \
 ./tools/dist_train.sh \
 configs/SiamSeg/siamseg_daformer_sepaspp_mitb5_512x512_40k_VaihingenIRRG_2_PotsdamRGB.py 4

3.6 LoveDA Rural to Urban

# Potsdam IRRG to Vaihingen IRRG
# CUDA_VISIBLE_DEVICES Visible GPU ids are 0-3 Total four GPU processors
# PORT Sets the communication port of the master for distributed training.
# The last 4 indicates the total number of GPUs used
CUDA_VISIBLE_DEVICES=0,1,2,3  PORT=10985 \
 ./tools/dist_train.sh \
 configs/SiamSeg/siamseg_daformer_sepaspp_mitb5_512x512_40k_Rural_2_Urban.py 4

4.Testing

4.1 Potsdam IRRG to Vaihingen IRRG

# for dist test
CUDA_VISIBLE_DEVICES=4,5,6,7  PORT=10985 \
  sh tools/dist_test.sh \
  configs/SiamSeg/siamseg_daformer_sepaspp_mitb5_512x512_40k_PotsdamRGB_2_VaihingenIRRG.py  \
  {beast_model_path}  4 --eval mIoU mFscore

# for predict label save  
# launcher must set to 'none'
# opacity between 0 and 1
PYTHONPATH=$(pwd):$PYTHONPATH  CUDA_VISIBLE_DEVICES=0 \
  python tools/test.py \
  configs/SiamSeg/siamseg_daformer_sepaspp_mitb5_512x512_40k_PotsdamRGB_2_VaihingenIRRG.py  \
  {beast_model_path} --eval mIoU mFscore --launcher none  --opacity 1.0

4.2 Potsdam RGB to Vaihingen IRRG

# for dist test
CUDA_VISIBLE_DEVICES=4,5,6,7 PORT=10985 \
  sh tools/dist_test.sh \
  configs/SiamSeg/siamseg_daformer_sepaspp_mitb5_512x512_40k_PotsdamRGB_2_VaihingenIRRG.py  \
  {beast_model_path}  4 --eval mIoU mFscore

# for predict label save  
# launcher must set to 'none'
# opacity between 0 and 1
PYTHONPATH=$(pwd):$PYTHONPATH  CUDA_VISIBLE_DEVICES=0 \
  python tools/test.py \
  configs/SiamSeg/siamseg_daformer_sepaspp_mitb5_512x512_40k_PotsdamRGB_2_VaihingenIRRG.py  \
  {beast_model_path} --eval mIoU mFscore --launcher none  --opacity 1.0

4.3 Vaihingen IRRG to Potsdam IRRG

# for dist test
CUDA_VISIBLE_DEVICES=4,5,6,7 PORT=10985 \
  sh tools/dist_test.sh \
  configs/SiamSeg/siamseg_daformer_sepaspp_mitb5_512x512_40k_VaihingenIRRG_2_PotsdamIRRG.py  \
  {beast_model_path}  4 --eval mIoU mFscore

# for predict label save  
# launcher must set to 'none'
# opacity between 0 and 1
PYTHONPATH=$(pwd):$PYTHONPATH  CUDA_VISIBLE_DEVICES=0 \
  python tools/test.py \
  configs/SiamSeg/siamseg_daformer_sepaspp_mitb5_512x512_40k_VaihingenIRRG_2_PotsdamIRRG.py  \
  {beast_model_path} --eval mIoU mFscore --launcher none  --opacity 1.0

4.4 Vaihingen IRRG to Potsdam RGB

# for dist test
CUDA_VISIBLE_DEVICES=4,5,6,7 PORT=10985 \
  sh tools/dist_test.sh \
  configs/SiamSeg/siamseg_daformer_sepaspp_mitb5_512x512_40k_VaihingenIRRG_2_PotsdamRGB.py  \
  {beast_model_path}  4 --eval mIoU mFscore
# for predict label save

# for predict label save  
# launcher must set to 'none'
# opacity between 0 and 1
PYTHONPATH=$(pwd):$PYTHONPATH  CUDA_VISIBLE_DEVICES=0 \
  python tools/test.py \
  configs/SiamSeg/siamseg_daformer_sepaspp_mitb5_512x512_40k_VaihingenIRRG_2_PotsdamRGB.py  \
  {beast_model_path} --eval mIoU mFscore --launcher none  --opacity 1.0

4.5 LoveDA Rural to Urban

# for dist test
CUDA_VISIBLE_DEVICES=4,5,6,7 PORT=10985 \
  sh tools/dist_test.sh \
  configs/SiamSeg/siamseg_daformer_sepaspp_mitb5_512x512_40k_Rural_2_Urban.py  \
  {beast_model_path}  4 --eval mIoU mFscore

# for predict label save  
# launcher must set to 'none'
# opacity between 0 and 1
PYTHONPATH=$(pwd):$PYTHONPATH  CUDA_VISIBLE_DEVICES=0 \
  python tools/test.py \
  configs/SiamSeg/siamseg_daformer_sepaspp_mitb5_512x512_40k_Rural_2_Urban.py  \
  {beast_model_path} --eval mIoU mFscore --launcher none  --opacity 1.0

References

Many thanks to their excellent works

woldier / SiamSeg

readme

SiamSeg: Self-Training with Contrastive Learning for Unsupervised Domain Adaptation Semantic Segmentation in Remote Sensing

News:

1. Creating Virtual Environment

2.Preparation of data sets

2.1 Download of datasets

ISPRS Potsdam

ISPRS Vaihingen

LoveDA

2.2 Data set preprocessing

3.Training

3.1 Preparation of pre-trained models

3.2 Potsdam IRRG to Vaihingen IRRG

3.3 Potsdam RGB to Vaihingen IRRG

3.4 Vaihingen IRRG to Potsdam IRRG

3.5 Vaihingen IRRG to Potsdam RGB

3.6 LoveDA Rural to Urban

4.Testing

4.1 Potsdam IRRG to Vaihingen IRRG

4.2 Potsdam RGB to Vaihingen IRRG

4.3 Vaihingen IRRG to Potsdam IRRG

4.4 Vaihingen IRRG to Potsdam RGB

4.5 LoveDA Rural to Urban

References