More detail for package version

[DTDwind]

Hello,

I wanna to reproduce this work more accurately.

May I get more details what you used?

Maybe is numpy version or others information?

[fnlandini]

Hello, From the environment where I usually ran these experiments I have numpy 1.19.4 scipy 1.4.1 sklearn 0.20.3 numexpr 2.6.9 h5py 2.9.0 onnxruntime 1.4.0 soundfile 0.10.3 torch 1.6.0 tabulate 0.8.6

In any case, a different version should not produce substantially different results. Differences of up to 0.2 DER can happen just from changing the random seed in the code so I would expect differences because of packages' versions to be even smaller.

[DTDwind]

@fnlandini Thanks for your answer, I will try it.

[Jamiroquai88]

@fnlandini we should probably update requirements.txt and setup.py with those versions

[fnlandini]

@Jamiroquai88 I agree, but I'd like to test that we obtain the same results we report with these particular packages to be 100% sure of them. I'm rather busy lately but I'll keep this issue open to remind me of doing so

[DTDwind]

Hello @fnlandini , my version environment may be the same as yours. Now, my VBx DER is 19.07. And AHC DER is 21.43! The AHC results are same as yours.

This is my AHC's result_full.

File               DER    JER    B3-Precision    B3-Recall    B3-F1    GKT(ref, sys)    GKT(sys, ref)    H(ref|sys)    H(sys|ref)    MI    NMI
---------------  -----  -----  --------------  -----------  -------  ---------------  ---------------  ------------  ------------  ----  -----
EN2002a          11.70  19.74            0.89         0.82     0.85             0.78             0.86          0.37          0.73  2.08   0.79
EN2002b          14.09  26.64            0.88         0.81     0.84             0.76             0.85          0.38          0.81  2.05   0.78
EN2002c           9.93  27.64            0.89         0.86     0.88             0.82             0.85          0.38          0.55  1.71   0.79
EN2002d          12.94  23.54            0.88         0.81     0.84             0.77             0.85          0.41          0.81  2.04   0.77
ES2004a           9.68  12.35            0.95         0.87     0.91             0.83             0.93          0.18          0.46  1.97   0.86
ES2004b           4.50   5.37            0.96         0.93     0.94             0.91             0.95          0.13          0.30  2.11   0.91
ES2004c           3.93   6.02            0.96         0.93     0.95             0.92             0.95          0.15          0.29  2.09   0.90
ES2004d           7.84  12.42            0.93         0.89     0.91             0.86             0.92          0.23          0.47  1.99   0.85
IS1009a           6.89  14.65            0.94         0.90     0.92             0.84             0.90          0.18          0.36  1.57   0.85
IS1009b           3.96   4.97            0.97         0.93     0.95             0.91             0.96          0.12          0.32  2.13   0.91
IS1009c           3.55   4.86            0.98         0.94     0.96             0.93             0.97          0.09          0.27  2.15   0.92
IS1009d           5.40   7.33            0.96         0.91     0.94             0.88             0.94          0.15          0.37  1.89   0.88
TS3003a          12.24  15.96            0.98         0.85     0.91             0.75             0.96          0.05          0.51  1.12   0.81
TS3003b           6.61   6.45            0.97         0.90     0.93             0.87             0.96          0.09          0.44  2.05   0.89
TS3003c           4.20   3.89            0.98         0.94     0.96             0.92             0.97          0.07          0.28  2.16   0.93
TS3003d           6.35  18.08            0.94         0.91     0.92             0.88             0.91          0.21          0.36  1.91   0.87
*** OVERALL ***   7.60  13.57            0.94         0.89     0.91             0.89             0.94          0.21          0.46  5.91   0.95

[fnlandini]

Updated requirements.txt and setup.py