English | 简体中文

OpenESS: Event-Based Semantic Scene Understanding with Open Vocabularies

Lingdong Kong^1,2    Youquan Liu³    Lai Xing Ng⁴    Benoit R. Cottereau^5,6    Wei Tsang Ooi^1
1National University of Singapore    ²CNRS@CREATE    ³Hochschule Bremerhaven    ⁴Institute for Infocomm Research, A*STAR    ⁵IPAL, CNRS IRL 2955, Singapore    ⁶CerCo, CNRS UMR 5549, Universite Toulouse III

About

OpenESS is an open-vocabulary event-based semantic segmentation (ESS) framework that synergizes information from image, text, and event-data domains to enable scalable ESS in an open-world, annotation-efficient manner.

Input Event Stream	“Driveable”	“Car”	“Manmade”
Zero-Shot ESS	“Walkable”	“Barrier”	“Flat”

Updates

• [2024.05] - Our paper is available on arXiv, click here to check it out. The code will be available later.
• [2024.04] - OpenESS was selected as a :sparkles: highlight :sparkles: at CVPR 2024 (2.8% = 324/11532).
• [2024.02] - OpenESS was accepted to CVPR 2024! :tada:

Outline

• Demo
• Installation
• Data Preparation
• Getting Started
• Benchmark
• TODO List
• Citation
• License
• Acknowledgements

:movie_camera: Demo

Demo #1	Demo #2	Demo #3
[YouTube]() :arrow_heading_up:	[YouTube]() :arrow_heading_up:	[YouTube]() :arrow_heading_up:

:gear: Installation

Kindly refer to INSTALL.md for the installation details.

:hotsprings: Data Preparation

Kindly refer to DATA_PREPARE.md for the details to prepare the [DDD17-Seg]() and [DSEC-Semantic]() datasets.

:rocket: Getting Started

Please refer to GET_STARTED.md to learn more about how to use this codebase.

:bar_chart: Benchmark

OpenESS Framework

Annotation-Free ESS

To be updated.

Fully-Supervised ESS

To be updated.

Open-Vocabulary ESS

To be updated.

Qualitative Assessment

:memo: TODO List

To be updated.

Citation

If you find this work helpful, please kindly consider citing our paper:

@inproceedings{kong2024openess,
  title = {OpenESS: Event-Based Semantic Scene Understanding with Open Vocabularies},
  author = {Kong, Lingdong and Liu, Youquan and Ng, Lai Xing and Cottereau, Benoit R. and Ooi, Wei Tsang},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2024},
}

License

This work is under the Apache License Version 2.0, while some specific implementations in this codebase might be with other licenses. Kindly refer to LICENSE.md for a more careful check, if you are using our code for commercial matters.

Acknowledgements

To be updated.