Awesome-Panoptic-Segmentation 
This repo is a collection of the challenging panoptic segmentation, including papers, codes, and benchmark results, etc.
Outline
Panoptic Segmentation
Summarize in one sentence : Panoptic Segmentation proposes to solve the semantic segmentation(Stuff) and instance segmentation(Thing) in a unified and general manner.
Structure Overview
from UPSNet.
Datasets
Generally, the datasets which contains both semantic and instance annotations can be used to solve the challenging panoptic task.
Evaluation
Metrics
-
PQare the standard metrics described in Panoptic Segmentation.
-
PCare the standard metrics described in DeeperLab.
Evaluation Code
Competition
- AutoNUE 2019 Panoptic Segmentation Challenge (ICCV 2019 Workshop, Closed)
- COCO 2019 Panoptic Segmentation Task (ICCV 2019 Workshop, Closed)
- Mapillary 2019 Panoptic Segmentation Task (ICCV 2019 Workshop, Closed)
- Cityscapes Panoptic Semantic Labeling Task (Open)
- COCO 2018 Panoptic Segmentation Task (ECCV 2018 Workshop, Closed)
- Mapillary Vistas 2018 Panoptic Segmentation Task (ECCV 2018 Workshop, Closed)
Benchmark Results
COCO val Benchmark
| Method | Backbone | PQ | PQ-Thing | PQ-Stuff | SQ | RQ | mIoU | AP-Mask | PC | e2e |
|---|---|---|---|---|---|---|---|---|---|---|
| SOGNet | ResNet-50 | 43.7 | 50.6 | 33.2 | 78.7 | 53.5 | 54.56 | 34.2 | - | :white_check_mark: |
| UPSNet | ResNet-50 | 42.5 | 48.6 | 33.4 | - | - | 54.3 | 34.3 | - | :white_check_mark: |
| OANet | ResNet-101 | 41.3 | 50.4 | 27.7 | - | - | - | - | - | :white_check_mark: |
| OCFusion | ResNet-50 | 41.0 | 49.0 | 29.0 | 77.1 | 50.6 | - | - | - | :white_check_mark: |
| Panoptic FPN | ResNet-101 | 40.9 | 48.3 | 29.7 | - | - | - | - | - | :white_check_mark: |
| AUNet | ResNet-50 | 39.6 | 49.1 | 25.2 | - | - | 45.1 | 34.7 | - | :white_check_mark: |
| AdaptIS | ResNet-101 | 37.0 | 41.8 | 29.9 | - | - | - | - | - | :white_check_mark: |
| DeeperLab | Xception-71 | 34.3 | 37.5 | 29.6 | 77.1 | 43.1 | - | - | 56.8 | :white_check_mark: |
Cityscapes valBenchmark
| Method | Backbone | PQ | PQ-Thing | PQ-Stuff | SQ | RQ | mIoU | AP-Mask | PC | e2e |
|---|---|---|---|---|---|---|---|---|---|---|
| Panoptic(Merge) | - | 61.2 | 66.4 | 54.0 | 80.9 | 74.4 | - | - | - | :negative_squared_cross_mark: |
| AdaptIS | ResNet-101 | 60.6 | 58.7 | 64.4 | - | - | 79.2 | 36.3 | - | :white_check_mark: |
| SOGNet | ResNet-50 | 60.0 | 56.7 | 62.5 | - | - | - | - | - | :white_check_mark: |
| Seamless | ResNet-50 | 59.8 | 53.4 | 64.5 | - | - | 75.4 | 31.9 | - | :white_check_mark: |
| UPSNet | ResNet-50 | 59.3 | 54.6 | 62.7 | 79.7 | 73.0 | 75.2 | 33.3 | - | :white_check_mark: |
| TASCNet | ResNet-101 | 59.2 | 56 | 61.5 | - | - | 77.8 | 37.6 | - | :white_check_mark: |
| AUNet | ResNet-101 | 59.0 | 54.8 | 62.1 | - | - | 75.6 | 34.4 | - | :white_check_mark: |
| Panoptic FPN | ResNet-101 | 58.1 | 52.0 | 62.5 | - | - | 75.7 | 33.0 | - | :white_check_mark: |
| DeeperLab | Xception-71 | 56.5 | - | - | - | - | - | - | 75.6 | :white_check_mark: |
Mapillary val Benchmark
| Method | Backbone | PQ | PQ-Thing | PQ-Stuff | SQ | RQ | mIoU | AP-Mask | PC | e2e |
|---|---|---|---|---|---|---|---|---|---|---|
| Panoptic(Merge) | - | 38.3 | 41.8 | 35.7 | 73.6 | 47.7 | - | - | - | :negative_squared_cross_mark: |
| Seamless | ResNet-50 | 37.2 | 33.2 | 42.5 | - | - | 50.2 | 16.3 | - | :white_check_mark: |
| AdaptIS | ResNet-101 | 33.4 | 28.3 | 40.3 | - | - | - | - | - | :white_check_mark: |
| TASCNet | ResNet-101 | 32.6 | 31.3 | 34.4 | - | - | 35.0 | 18.5 | - | :white_check_mark: |
| DeeperLab | Xception-71 | 32.0 | - | - | - | - | - | - | 55.3 | :white_check_mark: |
Papers
AAAI2020
- SOGNet: Yibo Yang, Hongyang Li, Xia Li, Qijie Zhao, Jianlong Wu, Zhouchen Lin.
"SOGNet: Scene Overlap Graph Network for Panoptic Segmentation." AAAI (2020). [paper]
ICCV2019
-
AdaptIS: Konstantin Sofiiuk, Olga Barinova, Anton Konushin.
"AdaptIS: Adaptive Instance Selection Network." ICCV (2019). [paper] -
Cheng-Yang Fu, Tamara L. Berg, Alexander C. Berg.
"IMP: Instance Mask Projection for High Accuracy Semantic Segmentation of Things." ICCV (2019). [paper] -
Bowen Cheng, Maxwell D. Collins, Yukun Zhu, Ting Liu, Thomas S. Huang, Hartwig Adam, Liang-Chieh Chen.
"Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation Bowen." ICCVW (2019). [paper]
CVPR2019
-
Panoptic Segmentation: Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, Piotr Dollár.
"Panoptic Segmentation." CVPR (2019). [paper] -
Panoptic FPN: Alexander Kirillov, Ross Girshick, Kaiming He, Piotr Dollár.
"Panoptic Feature Pyramid Networks." CVPR (2019 oral). [paper] [unofficial code][detectron2] -
AUNet: Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, Xingang Wang.
"Attention-guided Unified Network for Panoptic Segmentation." CVPR (2019). [paper] -
UPSNet: Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, Raquel Urtasun.
"UPSNet: A Unified Panoptic Segmentation Network." CVPR (2019 oral). [paper] [code] -
DeeperLab: Tien-Ju Yang, Maxwell D. Collins, Yukun Zhu, Jyh-Jing Hwang, Ting Liu, Xiao Zhang, Vivienne Sze, George Papandreou, Liang-Chieh Chen.
"DeeperLab: Single-Shot Image Parser." CVPR (2019 oral). [paper] [project] [code] -
OANet: Huanyu Liu, Chao Peng, Changqian Yu, Jingbo Wang, Xu Liu, Gang Yu, Wei Jiang.
"An End-to-End Network for Panoptic Segmentation." CVPR (2019). [paper] -
Eirikur Agustsson, Jasper R. R. Uijlings, Vittorio Ferrari .
"Interactive Full Image Segmentation by Considering All Regions Jointly." CVPR (2019). [paper] -
Seamless: Lorenzo Porzi, Samuel Rota Bulo, Aleksander Colovic, Peter Kontschieder.
"Seamless Scene Segmentation." CVPR (2019) (Extended Version). [paper][code]
ECCV2018
- Qizhu Li, Anurag Arnab, Philip H.S. Torr.
"Weakly- and Semi-Supervised Panoptic Segmentation." ECCV (2018). [paper] [code]
ArXiv
-
Rohit Mohan, Abhinav Valada.
"EfficientPS: Efficient Panoptic Segmentation." arXiv (2020). [paper] -
Rui Hou, Jie Li, Arjun Bhargava, Allan Raventos, Vitor Guizilini, Chao Fang, Jerome Lynch, Adrien Gaidon.
"Real-Time Panoptic Segmentation from Dense Detections." arXiv (2019). [paper] -
Mark Weber, Jonathon Luiten, Bastian Leibe.
"Single-Shot Panoptic Segmentation." arXiv (2019). [paper] -
Qiang Chen, Anda Cheng, Xiangyu He, Peisong Wang, Jian Cheng.
"SpatialFlow: Bridging All Tasks for Panoptic Segmentation." arXiv (2019). [paper] -
Sagi Eppel, Alan Aspuru-Guzik.
"Generator evaluator-selector net: a modular approach for panoptic segmentation." arXiv (2019). [paper] -
Jasper R. R. Uijlings, Mykhaylo Andriluka, Vittorio Ferrari.
"Panoptic Image Annotation with a Collaborative Assistant." arXiv (2019). [paper] -
OCFusion: Justin Lazarow, Kwonjoon Lee, Zhuowen Tu.
"Learning Instance Occlusion for Panoptic Segmentation." arXiv (2019). [paper] -
PEN: Yuan Hu, Yingtian Zou, Jiashi Feng.
"Panoptic Edge Detection." arXiv (2019). [paper] -
TASCNet: Jie Li, Allan Raventos, Arjun Bhargava, Takaaki Tagawa, Adrien Gaidon.
"Learning to Fuse Things and Stuff." arXiv (2018). [paper] -
Daan de Geus, Panagiotis Meletis, Gijs Dubbelman.
"Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network." arXiv (2018). [paper] -
Daan de Geus, Panagiotis Meletis, Gijs Dubbelman.
"Single Network Panoptic Segmentation for Street Scene Understanding." arXiv (2019). [paper] -
David Owen, Ping-Lin Chang.
"Detecting Reflections by Combining Semantic and Instance Segmentation." arXiv (2019). [paper] -
Gaku Narita, Takashi Seno, Tomoya Ishikawa, Yohsuke Kaji.
"PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things." arXiv (2019, IROS). [paper]
Tutorials
- CVPR 2019 Tutorial on Visual Recognition and Beyond. [slides] [homepage]
- COCO 2017 Workshop. [slides]
Blogs
- Megvii(Face++) Detection Team. [zhihu]