PR_AUC of mood/theme is 8.19 in Table 2 but is 0.0814 in results/auc_mood.tsv

[annahung31]

Hi, thanks for providing the result AUC of the baseline experiment. I found out that In the paper Table 2, the PR_AUC of mood/theme is 8.19, but when I take the average of PR_AUC of results/auc_mood.tsv , I got 0.0814.

Is there anything I miss? Thank you.

[rfalcon100]

Did you manage to fix this? I am getting the following error when computing the final scores. This is using the provided mel spectrograms.

Only one class present in y_true. ROC AUC score is not defined in that case.

[dbogdanov]

@rfalcon100 Have you checked you load the ground-truth correctly? It looks like you have only one tag instead of 57.

[rfalcon100]

@dbogdanov There seems to be some problems with the splits. For example, in the top50 set, there are some tags that have no samples. Modifying the get_npy.pyfile to display a count for all tags:

def get_npy_array(path, tag_list, option, type_='train'):
    ...

    aa = []
    for k, v in dictionary.items():
        aa.append(v['tags'])
    bb = np.array(aa)

    cc = bb.sum(axis=0)
    print(cc)

The count for all tags in the validation set, split 0, top50, is:

`[ 876. 1546. 401. 833. 1277. 605. 311. 918. 2921. 0. 696. 409.

808. 324. 518. 218. 631. 536. 224. 574. 542. 1621. 213. 434.
     325. 165. 1499. 1765. 313. 362. 510. 296. 410. 1450. 643. 562.
     326. 1093. 1083. 500. 1929. 0. 1719. 675. 502. 0. 233. 187.
809. 278.]`

So there is an error when computing the roc_auc score when the ground truth has tags with count 0. If I remove such columns when computing the scores, it works ok.

Am I doing something wrong?

[dbogdanov]

Adding @philtgun.

[minzwon]

@rfalcon100 It was the previous version of the split. I will update it with the current split. Thank you for the reporting.

[minzwon]

Updated the tag_list_50.npy as the newest version.

[rfalcon100]

Thanks, it seems to be working ok now.