sbintuitions / JMTEB

The evaluation scripts of JMTEB (Japanese Massive Text Embedding Benchmark)
Creative Commons Attribution Share Alike 4.0 International
31 stars 9 forks source link

[Fix] STSにおいてbfloat16 tensorが起こった不具合 #47

Closed lsz05 closed 3 months ago

lsz05 commented 3 months ago

関連する Issue / PR

N/A

PR をマージした後の挙動の変化

STSにscipyの関数を使っており,その内部にtensor.numpy()の処理があり,tensortorch.bfloat16タイプの場合,不具合が起こります。

[rank2]:   File "~/JMTEB/src/jmteb/evaluators/sts/evaluator.py", line 74, in __call__
[rank2]:     val_results[sim_name], _ = self._compute_similarity(
[rank2]:   File "~/JMTEB/src/jmteb/evaluators/sts/evaluator.py", line 112, in _compute_similarity
[rank2]:     pearson = pearsonr(golden_scores, sim_scores)[0]
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/scipy/stats/_stats_py.py", line 4727, in pearsonr
[rank2]:     y = np.asarray(y)
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/torch/_tensor.py", line 1087, in __array__
[rank2]:     return self.numpy()
[rank2]: TypeError: Got unsupported ScalarType BFloat16

挙動の変更を達成するために行ったこと

sim_scorestorch.bfloat16である場合,.float()dtype変更

動作確認

akiFQC commented 3 months ago

LGTM