search

fschmid56 / EfficientAT

This repository aims at providing efficient CNNs for Audio Tagging. We provide AudioSet pre-trained models ready for downstream training and extraction of audio embeddings.
MIT License
218 stars 41 forks source link

error in wandb report DyMN on OpenMic #29

Closed joris-vaneyghen closed 4 months ago

joris-vaneyghen commented 4 months ago

The documentation for fine-tuning DyMN on OpenMic includes a reported example run, accessible via the following link: https://api.wandb.ai/links/florians/qo32vrgl.

In this report, the mean Average Precision (mAP) is stated as 91.6, whereas the paper reports a mAP of 84.4. Interestingly, the mAP value I obtained from fine-tuning on my personal computer aligns closely with the paper's reported value.

Upon closer inspection, I noticed that all mAP values correspond exactly to the values of the ROC statistics. This raises the question: could incorrect statistics be logged to W&B?

fschmid56 commented 4 months ago

Hi, thanks for pointing this out. The results in the paper are correct and I think also the code in this repo. The report was messed up somehow. Here are the new reports (from experiments that are still running at the time of writing this answer):

MN10: https://api.wandb.ai/links/florians/a23ye448 DyMN-M: https://api.wandb.ai/links/florians/680sv79u

The mAP values should be correct now.