RuntimeWarning: invalid value encountered in log

[bhigy]

Hi @bshall ,

Still trying to replicate your results. I now get some scores, not exactly the same you report but close enough. However I also get warnings during the evaluation and I am wondering whether they might explain the small difference. Do you get anything like that:

/home/bjrhigy/opt/miniconda3/envs/zerospeech2020/lib/python3.8/site-packages/ABXpy/distances/metrics/kullback_leibler.py:15: RuntimeWarning: invalid value encountered in log
  pq = np.dot(x, np.log(y.transpose()))
/home/bjrhigy/opt/miniconda3/envs/zerospeech2020/lib/python3.8/site-packages/ABXpy/distances/metrics/kullback_leibler.py:17: RuntimeWarning: invalid value encountered in log
  np.sum(x * np.log(x), axis=1).reshape(x.shape[0], 1), (1, y.shape[0]))
/home/bjrhigy/opt/miniconda3/envs/zerospeech2020/lib/python3.8/site-packages/ABXpy/score.py:113: RuntimeWarning: invalid value encountered in less
  scores = (np.int8(dis_AX < dis_BX) -
/home/bjrhigy/opt/miniconda3/envs/zerospeech2020/lib/python3.8/site-packages/ABXpy/score.py:114: RuntimeWarning: invalid value encountered in greater
  np.int8(dis_AX > dis_BX))

The exact command I ran: zerospeech2020-evaluate 2019 submission/ -o eval.json -D ~/corpora/zerospeech2020

[bshall]

Hi @bhigy,

Would you mind posting your results? I also get those warnings but never bothered to track the issue down. One thing that could potentially make a small difference is the rounding here. Otherwise, it could be differences in the cuda version? Hopefully, the scores are close enough that its relatively negligible?

The exact command I ran: zerospeech2020-evaluate 2019 submission/ -o eval.json -D ~/corpora/zerospeech2020

Yeah, that's essentially what I ran too.

[bhigy]

It is very negligeable and only for the ABX score with cosine. I will consider this as a success! :)

{
    "2019": {
        "english": {
            "scores": {
                "abx": 14.043385117056495,
                "bitrate": 412.2387509949519
            },
            "details_bitrate": {
                "test": 412.2387509949519
            },
            "details_abx": {
                "test": {
                    "cosine": 14.043385117056495,
                    "KL": 50.0,
                    "levenshtein": 35.927825062038984
                }
            }
        }
    }
}