Update on 2023/11: SePiCo is selected as :trophy: ESI Highly Cited Paper!!
Update on 2023/02/15: Code release for Cityscapes → Dark Zurich.
Update on 2023/01/14: π₯³ We are happy to announce that SePiCo has been accepted in an upcoming issue of the TPAMI.
Update on 2022/09/24: All checkpoints are available.
Update on 2022/09/04: Code release.
Update on 2022/04/20: ArXiv Version of SePiCo is available.
//: # ()
//: # ()
//: # ()
In this work, we propose Semantic-Guided Pixel Contrast (SePiCo), a novel one-stage adaptation framework that highlights the semantic concepts of individual pixel to promote learning of class-discriminative and class-balanced pixel embedding space across domains, eventually boosting the performance of self-training methods.
This code is implemented with Python 3.8.5
and PyTorch 1.7.1
on CUDA 11.0
.
To try out this project, it is recommended to set up a virtual environment first:
# create and activate the environment
conda create --name sepico -y python=3.8.5
conda activate sepico
# install the right pip and dependencies for the fresh python
conda install -y ipython pip
Then, the dependencies can be installed by:
# install required packages
pip install -r requirements.txt
# install mmcv-full, this command compiles mmcv locally and may take some time
pip install mmcv-full==1.3.7 # requires other packeges to be installed first
Alternatively, the mmcv-full
package can be installed faster with official pre-built packages, for instance:
# another way to install mmcv-full, faster
pip install mmcv-full==1.3.7 -f https://download.openmmlab.com/mmcv/dist/cu110/torch1.7.0/index.html
The environment is now fully prepared.
Symlink the required datasets:
ln -s /path/to/gta5/dataset data/gta
ln -s /path/to/cityscapes/dataset data/cityscapes
ln -s /path/to/dark_zurich/dataset data/dark_zurich
Perform preprocessing to convert label IDs to the train IDs and gather dataset statistics:
python tools/convert_datasets/gta.py data/gta --nproc 8
python tools/convert_datasets/cityscapes.py data/cityscapes --nproc 8
Ultimately, the data structure should look like this:
SePiCo
βββ ...
βββ data
β βββ cityscapes
β β βββ gtFine
β β βββ leftImg8bit
β βββ dark_zurich
β β βββ corresp
β β βββ gt
β β βββ rgb_anon
β βββ gta
β β βββ images
β β βββ labels
βββ ...
We provide pretrained models of both Domain Adaptive Semantic Segmentation tasks through Google Drive and Baidu Netdisk (access code: pico
).
variants | model name | mIoU | checkpoint download |
---|---|---|---|
DistCL | sepico_distcl_gta2city_dlv2.pth | 61.0 | Google / Baidu (acc: pico ) |
BankCL | sepico_bankcl_gta2city_dlv2.pth | 59.8 | Google / Baidu (acc: pico ) |
ProtoCL | sepico_protocl_gta2city_dlv2.pth | 58.8 | Google / Baidu (acc: pico ) |
variants | model name | mIoU | checkpoint download |
---|---|---|---|
DistCL | sepico_distcl_gta2city_daformer.pth | 70.3 | Google / Baidu (acc: pico ) |
BankCL | sepico_bankcl_gta2city_daformer.pth | 68.7 | Google / Baidu (acc: pico ) |
ProtoCL | sepico_protocl_gta2city_daformer.pth | 68.5 | Google / Baidu (acc: pico ) |
variants | model name | mIoU | checkpoint download |
---|---|---|---|
DistCL | sepico_distcl_syn2city_dlv2.pth | 58.1 | Google / Baidu (acc: pico ) |
BankCL | sepico_bankcl_syn2city_dlv2.pth | 57.4 | Google / Baidu (acc: pico ) |
ProtoCL | sepico_protocl_syn2city_dlv2.pth | 56.8 | Google / Baidu (acc: pico ) |
variants | model name | mIoU | checkpoint download |
---|---|---|---|
DistCL | sepico_distcl_syn2city_daformer.pth | 64.3 | Google / Baidu (acc: pico ) |
BankCL | sepico_bankcl_syn2city_daformer.pth | 63.3 | Google / Baidu (acc: pico ) |
ProtoCL | sepico_protocl_syn2city_daformer.pth | 62.9 | Google / Baidu (acc: pico ) |
variants | model name | mIoU | checkpoint download |
---|---|---|---|
DistCL | sepico_distcl_city2dark_dlv2.pth | 45.4 | Google / Baidu (acc: pico ) |
BankCL | sepico_bankcl_city2dark_dlv2.pth | 44.1 | Google / Baidu (acc: pico ) |
ProtoCL | sepico_protocl_city2dark_dlv2.pth | 42.6 | Google / Baidu (acc: pico ) |
variants | model name | mIoU | checkpoint download |
---|---|---|---|
DistCL | sepico_distcl_city2dark_daformer.pth | 54.2 | Google / Baidu (acc: pico ) |
BankCL | sepico_distcl_city2dark_daformer.pth | 53.3 | Google / Baidu (acc: pico ) |
ProtoCL | sepico_distcl_city2dark_daformer.pth | 52.7 | Google / Baidu (acc: pico ) |
Our trained model (sepico_distcl_city2dark_daformer.pth) is also tested for generalization on the Nighttime Driving and BDD100k-night test sets.
Method | model name | Dark Zurich-test | Nighttime Driving | BDD100k-night | checkpoint download |
---|---|---|---|---|---|
SePiCo | sepico_distcl_city2dark_daformer.pth | 54.2 | 56.9 | 40.6 | Google / Baidu (acc: pico ) |
To evaluate the pretrained models on Cityscapes, please run as follows:
python -m tools.test /path/to/config /path/to/checkpoint --eval mIoU
To evaluate on Dark Zurich, please get label predictions as follows and submit them to the official test server.
Get label predictions for the test set locally:
python -m tools.test /path/to/config /path/to/checkpoint --format-only --eval-options imgfile_prefix=/path/to/labelTrainIds
Note that the test server only accepts submission with the following directory structure:
submit.zip
βββ confidence
βββ labelTrainIds
βββ labelTrainIds_invalid
So we need to construct the confidence
and labelTrainIds_invalid
directory by hand (as they are not necessary to SePiCo evaluation).
Our practice is listed below for reference (check the example above for directory name):
cd dark_test/distcl_daformer
cp -r labelTrainIds labelTrainIds_invalid
cp -r labelTrainIds confidence
zip -q -r sepico_distcl_city2dark_daformer.zip labelTrainIds labelTrainIds_invalid confidence
# Now submit sepico_distcl_city2dark_daformer.zip to the test server for results.
To begin with, download SegFormer's official MiT-B5 weights (i.e., mit_b5.pth
) pretrained on ImageNet-1k from here and put it into a new folder ./pretrained
.
The training entrance is at run_experiments.py
. To examine the setting for a specific task, please take a look at experiments.py
for more details. Generally, the training script is given as:
python run_experiments.py --exp <exp_id>
Tasks 1~6 are run on GTAV → Cityscapes, and the mapping between <exp_id>
and tasks is:
<exp_id> |
variant | backbone | feature |
---|---|---|---|
1 |
DistCL | ResNet-101 | layer-4 |
2 |
BankCL | ResNet-101 | layer-4 |
3 |
ProtoCL | ResNet-101 | layer-4 |
4 |
DistCL | MiT-B5 | all-fusion |
5 |
BankCL | MiT-B5 | all-fusion |
6 |
ProtoCL | MiT-B5 | all-fusion |
Tasks 7~8 are run on Cityscapes → Dark Zurich, and the mapping between <exp_id>
and tasks is:
<exp_id> |
variant | backbone | feature |
---|---|---|---|
7 |
DistCL | ResNet-101 | layer-4 |
8 |
DistCL | MiT-B5 | all-fusion |
After training, the models can be tested following SePiCo Evaluation. Note that the training results are located in ./work_dirs
. The config filename should look like: 220827_1906_dlv2_proj_r101v1c_sepico_DistCL-reg-w1.0-start-iter3000-tau100.0-l3-w1.0_rcs0.01_cpl_self_adamw_6e-05_pmT_poly10warm_1x2_40k_gta2cs_seed76_4cc9a.json
, and the model file has suffix .pth
.
RandomCrop
class in mmseg/models/utils/ours_transforms.py.mmseg/models/uda/sepico.py
.This project is based on the following open-source projects. We thank their authors for making the source code publicly available.
If you find our work helpful, please starπ this repo and citeπ our paper. Thanks for your support!
@article{xie2023sepico,
title={Sepico: Semantic-guided pixel contrast for domain adaptive semantic segmentation},
author={Xie, Binhui and Li, Shuang and Li, Mingjia and Liu, Chi Harold and Huang, Gao and Wang, Guoren},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
year={2023},
publisher={IEEE}
}
For help and issues associated with SePiCo, or reporting a bug, please open a [GitHub Issues], or feel free to contact binhuixie@bit.edu.cn.