search

srvk / DiViMe

ACLEW Diarization Virtual Machine
Apache License 2.0
32 stars 9 forks source link

evaluation results: lack of pair? #155

Closed AlicjaRad closed 5 years ago

AlicjaRad commented 5 years ago

Hi,

I was comparing the test results to results pasted by in #132 by Marisa. Her output ended like this:

"Evaluation pipeline:

Evaluating noisemesSad on /vagrant/data/VanDam-Daylong/BN32/test with respect to : accuracy

Pairs that have been found : BN32_010007_test.rttm / noisemesSad_BN32_010007_test.rttm

accuracy report detection accuracy true positive true negative false positive false negative %
item
BN32_010007_test.rttm 11.24 30.80 3.11 0.20 267.57 TOTAL 11.24 30.80 3.11 0.20 267.57 accuracy report detection accuracy true positive true negative false positive false %
item
BN32_010007_test.rttm 11.24 30.80 3.11 0.20 267.57 TOTAL 11.24 30.80 3.11 0.20 267.57 Connection to 127.0.0.1 closed."

My test output end like this:

"Evaluation pipeline YOURS:

Evaluating noisemesSad on /vagrant/data/VanDam-Daylong/BN32/test with respect to : accuracy

Evaluation pipeline REFERENCE: detection accuracy true positive true negative false positive false %
item
BN32_010007_test.rttm 11.24 30.80 3.11 0.20 267.57 TOTAL 11.24 30.80 3.11 0.20 267.57 Connection to 127.0.0.1 closed."

It lacks "Pairs that have been found" and one set of results.

I attach also a file with output of running vagrant up.

I work on MacOS Mojave 10.14.6, 8GB RAM, 256 GB memory; I installed Vagrant 2.2.4 and Virtual Box 6.0.14

Vagrant_test.txt

MarvinLvn commented 5 years ago

Hello,

This is surprising, because in the latest version of the DiViME repo, the line responsible for logging pairs that have been found hasn't been commented out. Could you try to git pull the DiViMe repo to check if you're working with the latest version ?

Apart from that, those are just log lines. what matters is that the results are the same (which seems to be the case from the lines you showed us).

Concerning the missing set of results. It is possible that, before, the same set was displayed two times. Since you ran the valuation with one metric, you should have only one set of results

AlicjaRad commented 5 years ago

I installed everything today so I guess DiViMe is up to date. And git pull says that :)

Thus no reason for concern so far?

MarvinLvn commented 5 years ago

Yes, indeed. As long as the results are the same, everything's good ! I'll check but I think it might come from small log improvements to make the whole thing a bit clearer and more lightweight !

I'll close this issue for now :)