ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds

This is the official implementation of our paper ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds.

Training data, and checkpoints can be downloaded from ReCLAP's Google Drive.

Setup 🏋️

conda create -n reclap python=3.10
conda activate reclap
cd ReCLAP
pip install -r requirements.txt

Training 🏃‍♂️

Update the path to training csv (--train-data) validation csv (--val-data) and pretrained HTSAT checkpoint (--pretrained-audio) in run.sh file. The csvs can be downloaded from ReCLAP's Google Drive.

To run the training:

cd ReCLAP/train/src/laion_clap
sh run.sh

The checkpoint for CLAP 2.3M (tiny) and ReCLAP Base can be donwloaded from ReCLAP's Google Drive. The CLAP 2.3M (base) will be uploaded soon.

Prompt Augmentations

More details on prompt augemntation can be found in prompts.

Acknowledgement 🌻

We would like to thank the authors of LAION-CLAP for open-sourcing their code, which inspired our work.