[X] I have searched the existing issues and did not find a match.
Who can help?
No response
What are you working on?
As part of a dissertation, I am attempting to use cosine similarity to compare embeddings in order to assess the quality of machine translated text. Specifically I want to compare pairs of Russian original and English translated sentences. I am trialling various models for this including XLMRoberta. I am running this in Google Colab.
Current Behavior
The difference in cosine similarity between a pair of sentences with a validated, good quality translation and a translation which has no relation to the original at all is currently negligible.
Expected Behavior
Cosine similarity for a good translation is high and low for a poor one.
Is there an existing issue for this?
Who can help?
No response
What are you working on?
As part of a dissertation, I am attempting to use cosine similarity to compare embeddings in order to assess the quality of machine translated text. Specifically I want to compare pairs of Russian original and English translated sentences. I am trialling various models for this including XLMRoberta. I am running this in Google Colab.
Current Behavior
The difference in cosine similarity between a pair of sentences with a validated, good quality translation and a translation which has no relation to the original at all is currently negligible.
Expected Behavior
Cosine similarity for a good translation is high and low for a poor one.
Steps To Reproduce
https://colab.research.google.com/drive/10rtJmAkDgZLp5sVHsmeqMPZpKQ4NUkfu?usp=drive_link
Spark NLP version and Apache Spark
See notebook
Type of Spark Application
Python Application
Java Version
No response
Java Home Directory
No response
Setup and installation
See notebook
Operating System and Version
No response
Link to your project (if available)
No response
Additional Information
No response