search

MastafaF / multilingual_similarity_compare

Other
5 stars 0 forks source link

Test XLM-R with mean pooling after solving issue on mean pooling #8

Open MastafaF opened 4 years ago

MastafaF commented 4 years ago

Input:

sh similarity_XLM-R_batch.sh 40 mean True

Output: 

Confusion matrix:
langs   cs       de       en       es       fr       avg     
cs     0.00%   91.31%   97.64%   98.37%   94.34%   95.41%
de    93.84%    0.00%   88.05%   92.87%   95.40%   92.54%
en    91.21%   77.79%    0.00%   72.56%   94.77%   84.08%
es    95.47%   93.14%   61.84%    0.00%   90.28%   85.18%
fr    91.97%   81.25%   77.12%   71.96%    0.00%   80.58%
avg   93.12%   85.87%   81.16%   83.94%   93.70%   87.56%

Input: 

sh similarity_XLM-R_batch.sh 100 mean True

Output: 

Confusion matrix:
langs   cs       de       en       es       fr       avg     
cs     0.00%   91.24%   97.77%   97.80%   92.11%   94.73%
de    94.31%    0.00%   86.55%   92.44%   94.67%   91.99%
en    91.08%   74.93%    0.00%   71.56%   77.69%   78.81%
es    95.37%   88.38%   58.54%    0.00%   90.38%   83.17%
fr    92.71%   91.58%   87.41%   89.14%    0.00%   90.21%
avg   93.36%   86.53%   82.57%   87.74%   88.71%   87.78%
MastafaF commented 4 years ago

Comparison with CLS embedding for test purpose:

Input: 

sh similarity_XLM-R_batch.sh 40 cls True

Output:

Confusion matrix:
langs   cs       de       en       es       fr       avg     
cs     0.00%   97.60%   98.77%   99.53%   98.43%   98.58%
de    97.10%    0.00%   95.64%   97.44%   97.74%   96.98%
en    97.24%   95.94%    0.00%   94.14%   95.90%   95.80%
es    97.14%   97.44%   89.81%    0.00%   95.80%   95.05%
fr    97.20%   96.67%   91.18%   92.07%    0.00%   94.28%
avg   97.17%   96.91%   93.85%   95.80%   96.97%   96.14%

Input: 

sh similarity_XLM-R_batch.sh 100 cls True

Output: 

Confusion matrix:
langs   cs       de       en       es       fr       avg     
cs     0.00%   97.54%   98.83%   99.40%   98.57%   98.58%
de    97.04%    0.00%   95.50%   97.20%   97.44%   96.79%
en    97.20%   95.30%    0.00%   93.94%   94.34%   95.20%
es    96.97%   95.94%   90.71%    0.00%   94.81%   94.61%
fr    97.10%   97.10%   92.77%   93.64%    0.00%   95.15%
avg   97.08%   96.47%   94.46%   96.05%   96.29%   96.07%