search

The-Shuai / DeIL

Python code to implement DeIL, a CLIP based approach for open-world few-shot learning.
8 stars 0 forks source link

readme

DeIL

This paper has been accepted by CVPR 2024. We are profoundly grateful for the significant insights provided by CLIPN.

Abstract

Open-World Few-Shot Learning (OFSL) is a critical field of research, concentrating on the precise identification of target samples in environments with scarce data and unreliable labels, thus possessing substantial practical significance. Recently, the evolution of foundation models like CLIP has revealed their strong capacity for representation, even in settings with restricted resources and data. This development has led to a significant shift in focus, transitioning from the traditional method of “building models from scratch” to a strategy centered on “efficiently utilizing the capabilities of foundation models to extract relevant prior knowledge tailored for OFSL and apply it judiciously”. Amidst this backdrop, we unveil the Direct-and-Inverse CLIP (DeIL), an innovative method leveraging our proposed “Direct-and-Inverse” concept to activate CLIP-based methods for addressing OFSL. This concept transforms conventional single-step classification into a nuanced two-stage process: initially filtering out less probable categories, followed by accurately determining the specific category of samples. DeIL comprises two key components: a pre-trainer (frozen) for data denoising, and an adapter (tunable) for achieving precise final classification.

图片1

Get Started

  1. Create a conda environment and install dependencies. 
    pip install -r requirements.txt
  2. Download the "caches", "clipn_caches", "gpt_file" folders, and place them in the root directory.
  3. Download the CLIPN pre-trained model and place it in the "clipn_caches" directory.
    e.g., "./clipn_cache/CLIPN_ATD_Repeat2_epoch_10.pt".
    (please refer to CLIPN for more details)
  4. Follow Download_OFSL_Datasets.md to download the datasets.
  5. Modify the main_path in the main.py file on line 26 to match the dataset you intend to validate.
    e.g., set the main_path to main_path = "./configs/imagenet/config.yaml"
  6. Modify the root_path on the 2nd line of the config.yaml file corresponding to your dataset.
    e.g., within the ./configs/imagenet/config.yaml file, update the root_path to root_path: "./DATA/"
  7. Run 
    CUDA_VISIBLE_DEVICES=0 python main.py

    Acknowledgement

    This repo benefits from CLIPN, CaFo, CLIP, and DALL-E. Thanks for their wonderful work.

    Citation

    
@inproceedings{shao2024deil,
title={DeIL: Direct-and-Inverse CLIP for Open-World Few-Shot Learning},
author={Shao, Shuai and Bai, Yu and Wang, Yan and Liu, Baodi and Zhou, Yicong},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
pages={28505--28514},
year={2024}
}

@inproceedings{shao2024collaborative, title={Collaborative Consortium of Foundation Models for Open-World Few-Shot Learning}, author={Shao, Shuai and Bai, Yu and Wang, Yan and Liu, Baodi and Liu, Bin}, booktitle={Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)}, volume={38}, number={5}, pages={4740--4747}, year={2024} }