search

UKPLab / sentence-transformers

Multilingual Sentence & Image Embeddings with BERT
https://www.SBERT.net
Apache License 2.0
14.72k stars 2.43k forks source link

How to use LabelAccuracyEvaluator.py #26

Open solo0409 opened 4 years ago

solo0409 commented 4 years ago

Hi, I'd like to evaluate the accuracy of AllNLI task during training. Could you kindly provide an example for that? Thank you!

nreimers commented 4 years ago

Hi @solo0409 See issue #27 . The LabelAccuracyEvaluator.py as it is currently available (version 0.2.3 and lower) has a bug, it has not access to the softmax model. It will be fixed in version 0.2.4.

In #27, I also posted an example how to use this evaluator.

solo0409 commented 4 years ago

Hi @nreimers Thank you for your reply. Now it works!

solo0409 commented 4 years ago

Hi @nreimers

I tried to use replace the EmbeddingSimilarityEvaluator by LabelAccuracyEvaluator and evaluate model on NLI dev (use evaluation_stsbenchmark.py). However, I got a RuntimeError: Expected object of backend CPU but got backend CUDA for argument. Do you have a suggestion? Thanks

nreimers commented 4 years ago

Hi @solo0409 Thanks for pointing this out. I think a fix is described in this issue: https://github.com/UKPLab/sentence-transformers/issues/27#issuecomment-534379544

Will test it as soon as I have some more time and push it to the branch.