Open asnota opened 1 year ago
I've just tested BERTscore via HuggingFace datasets API, torchmetrics API and initial implementation, getting the same error. Therefore, BERTscore issue is not for GEM-metrics team. The issue with the Docker for BLEURT and Prism, hovewer, is.
@asnota I'm sorry for the delay. I'm afraid any metrics that require the use of a docker container just won't work with Colab. @danieldeutsch can you please confirm?
Hello, I've tried to test BERTscore, BLEURT and Prism embedding metrics from Colab, however got a stack or errors - the errors for BLEURT and Prism come from the same Docker source:
DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
,while BERTscore seems to have troubles with the dependencies:
No module named 'transformers.models.beit.configuration_beit'
Before trying the metrics, I did the heavy install, imported the library and initialized predictions and references objects:
The errors stack for BERTscore is as follows: