# Multimodal Cross-Domain Few-Shot Learning for Egocentric Action Recognition (ECCV'24) **[[Paper](https://masashi-hatano.github.io/assets/pdf/mm-cdfsl.pdf)][[Supplementary](https://masashi-hatano.github.io/assets/pdf/mm-cdfsl_supp.pdf)][[Project Page](https://masashi-hatano.github.io/MM-CDFSL/)][[Poster](https://masashi-hatano.github.io/assets/pdf/mm-cdfsl_poster.pdf)][[Data](https://huggingface.co/datasets/masashi-hatano/MM-CDFSL/tree/main)]**

This is the official code release for our ECCV 2024 paper \ "Multimodal Cross-Domain Few-Shot Learning for Egocentric Action Recognition".

🔨 Installation

# Create a virtual environment
python3 -m venv mm-cdfsl
source mm-cdfsl/bin/activate

# Install the dependencies
pip install -r requirements.txt

📂 Data Preparation

Training Split

You can find the train/val split files for all three target datasets in cdfsl folders.

Data Structure

Please follow the data structure as detailed in DATA_STRUCTURE.md.

Pre-processed Data

You can download the pre-processed data from the hub.

📍 Model Zoo

You can brouse the checkpoints of pre-trained model, comparison methods, and our models in this folder or directly download from the following links.

Pre-Train

2nd Stage

🔥 Training

1. Pre-training

Please make sure that you set modality (e.g., rgb) and dataset (e.g., epic) in configs/trainer/pretrain_trainer.yaml and confings/data_module/pretrain_data_module.yaml.

python3 lit_main_pretrain.py train=True test=False

2. Multimodal Distillation

Please make sure that you set dataset (e.g., epic) in confings/data_module/mm_distill_data_module.yaml. Also, you need to set the ckpt path of all modalities in configs/trainer/mm_distill_trainer.yaml.

python3 lit_main_mmdistill.py train=True test=False 

🔍 Evaluation

To evaluate the model in 5-way 5-shot setting with 600 runs, please run the following command.

python3 lit_main_mmdistill.py train=False test=True data_module.n_way=5 data_module.k_shot=5 data_module.episodes=600

✍️ Citation

If you use this code for your research, please cite our paper.

@inproceedings{Hatano2024MMCDFSL,
  author = {Hatano, Masashi and Hachiuma, Ryo and Fujii, Ryo and Saito, Hideo},
  title = {Multimodal Cross-Domain Few-Shot Learning for Egocentric Action Recognition},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year = {2024},
}