mimbres
commented
2 years ago Hi @Novicei, thanks for good questions (sorry for the late reply). Actually I am interested in improving the evaluation method, too.
I would like to know if the 500 test data you mentioned in your paper were randomly selected?Did you choose it only once? Or take the average of multiple times? Is your database set up to ensure that it contains 500 test data?
Selecting 500 test-songs was fixed (available in dataset release; isolated from train-songs). 2,000 test-queries (test_ids) were fixed, selected from the 500 test-songs. So they were randomly sampled only once.
- Pros: It was simple and useful at the early stage of development without a fast search algorithm.
- Cons: Lack of statistical significance. Possible improvement would be random-testing N times with large N and displaying mean (+- std. deviation) %.
Because in general, in the audio query task, there should be a difference between the recall rate and the precision rate. I understand that the accuracy rate in your paper is the precision rate.
The evaluation metric I used was Top1 hit rate(%) defined as:
In our setup, every query always has a single relevant original segment or sequence. Our DB is constructed as {test_500 + dummy_100K}.
As you mentioned, this can be thought of as Precision@1 where FP (false-positives) is # of retrieved irrelevant segments. Going deeper, it depends on how to define threshold. Assuming rank-threshold, it is exactly P@1. Furthermore, it would be possible to define a certain type of threshold (e.g. prob > 0.5) to calculate precision and recall.
I'm using a program implemented by someone else in pytorch, have you checked the program's implementation and results? When I tested, I found that the accuracy of 1-second and 2-second query tests fluctuated greatly with different data. Did you find this out?
I read the code, but haven't run it yet. But yes. The strong fluctuation may happen in my implementation, too.
And I think that there is also the possibility of muted segments for 1-second and 2-second queries, which will have a greater impact on the results. Therefore, your 1-second and 2-second accuracy rates are as stated in your paper. Is it right? Or is it more volatile?
Regardless of 1s or 2s input, I agree that such a silence or highly repetitive segment can greatly affect the test accuracy.
- In fact, the majority of genres in FMA dataset is experimental music (~50%) which contains a lot of such cases. One evidence is that I have experienced larger than 7% boost of performance while reproducing it with the updated testset.
unstable result: Unfortunately,the reproduced results (including this repo) are different from the paper's report. This inconsistency comes from the selection of test set and other reasons as discussed in #18. I might consider writing a note to account for this issue.1s vs 2s: I haven't tried2syet with this repo. We can discuss the issue after seeing the '2s' result based on this repo. I will share the result soon.
Novicei
stdio2016
Hello, I would like to know if the 500 test data you mentioned in your paper were randomly selected?Did you choose it only once? Or take the average of multiple times? Is your database set up to ensure that it contains 500 test data?Because in general, in the audio query task, there should be a difference between the recall rate and the precision rate. I understand that the accuracy rate in your paper is the precision rate. I'm using a program implemented by someone else in pytorch, have you checked the program's implementation and results? When I tested, I found that the accuracy of 1-second and 2-second query tests fluctuated greatly with different data. Did you find this out? And I think that there is also the possibility of muted segments for 1-second and 2-second queries, which will have a greater impact on the results. Therefore, your 1-second and 2-second accuracy rates are as stated in your paper. Is it right? Or is it more volatile?