Evaluation Metrics

[darius522]

Hi - The current code does not seem to cover the proposed evaluation portion. Would the authors potentially consider sharing their evaluation pipeline ? More specifically the implementations behind AIS, IIS, and AIC could be extremely helpful for reproducibility purpose.

Thank you! Darius

[BurakCanBiner]

Hello,

I pushed two files that you can use as a sample evaluation script. Please be aware that there are some parts that require local paths, some local files ( e.g. labels file) and can be further optimized. So this script will not work for you as is. Here is a breakdown of this script to make things clear:

— Step 1

We first read following files: Audio samples that we used to generate images, Images corresponding to these audios from the dataset Generated images with these datasets (possibly from multiple seeds) Labels of the dataset

— Step 2 For our generated images we create a properties dictionary to hold audio class, index of this class when sorted, audio path etc.

— Step 3 We calculate embeddings for several things ( audios, generated images, and paired dataset images) We load pre-calculated embeddings for entire dataset images to save time (e.g. “metrics_tensors/landscape_imgs.pt” inside the code) For your own dataset, you might as well need to pre-cache these.

— Step 4 For AIS and IIS we calculate two similarities

AIS - Using Wav2clip Generated images - ground truth audio embeddings ( lets call this matched similarities) Generated images - entire validations set audio embeddings

IIS - Using CLIP Generated images - paired images of ground truth audio embeddings ( lets call this matched similarities) Generated images - entire validations set image embeddings

— Step 5 For each image we count the number of the matched similarities are greater than the similarity of that image to entire set.

Since AIC is rather straightforward, I am not delving into details. You can refer to supplementary to see explanations. Once again please take this implementation only as a reference and do yours. I am hoping this script will help you in your research.

Sorry for the inconvenience so far, I want to turn this repository into a complete repository. But this work is still under review and there might be several further changes to this work. So that I am hesitant to do all the rearrangements just yet.