search

UKPLab / sentence-transformers

State-of-the-Art Text Embeddings
https://www.sbert.net
Apache License 2.0
15.45k stars 2.5k forks source link

the results of distillation of multilingual models is poor #1068

Open ScottishFold007 opened 3 years ago

ScottishFold007 commented 3 years ago

Inspired by < Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation > I expanded the original English-only model to support Chinese,I use the script - make_multilingual.py，and modified some settings,but I got poorer results, and here are some key training codes: image image image image image Accuracy src2trg and Accuracy trg2src was getting increasingly smaller ，and embedding similarity has deteriorating results...

So I want ask you - 1、What are the possible reasons for the above situation? 2、Can you tell me how you train ‘paraphrase-multilingual-mpnet-base-v2’？It works really well on Chinese！！！

Thank you in advance！

nreimers commented 3 years ago

The same script was used to train the multilingual models. From your screenshot I cannot see an issue.

Sometimes restarting helps. Transformer networks are known to diverge in some cases. Simply restarting helps.

ScottishFold007 commented 3 years ago

The same script was used to train the multilingual models. From your screenshot I cannot see an issue.

Sometimes restarting helps. Transformer networks are known to diverge in some cases. Simply restarting helps.

Thank you, nreimers. But I've restarted many times with the same result, which frustrates me and I really don't know what the problem is...

2021-07-14 12:49:11 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 50 steps: 2021-07-14 12:49:14 - Cosine-Similarity : Pearson: 0.4323 Spearman: 0.5337 2021-07-14 12:49:14 - Manhattan-Distance: Pearson: 0.5806 Spearman: 0.6003 2021-07-14 12:49:14 - Euclidean-Distance: Pearson: 0.4923 Spearman: 0.5314 2021-07-14 12:49:14 - Dot-Product-Similarity: Pearson: 0.1444 Spearman: 0.1586 2021-07-14 12:49:14 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 50 steps: 2021-07-14 12:49:16 - Accuracy src2trg: 36.50 2021-07-14 12:49:16 - Accuracy trg2src: 47.90 2021-07-14 12:49:16 - Save model to output/make-multilingual-en-zh_cn-zh_tw-2021-07-14_12-48-15 2021-07-14 12:49:47 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 100 steps: 2021-07-14 12:49:49 - Cosine-Similarity : Pearson: 0.2895 Spearman: 0.5137 2021-07-14 12:49:49 - Manhattan-Distance: Pearson: 0.5204 Spearman: 0.5824 2021-07-14 12:49:49 - Euclidean-Distance: Pearson: 0.4024 Spearman: 0.5099 2021-07-14 12:49:49 - Dot-Product-Similarity: Pearson: 0.1232 Spearman: 0.1686 2021-07-14 12:49:49 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 100 steps: 2021-07-14 12:49:51 - Accuracy src2trg: 40.20 2021-07-14 12:49:51 - Accuracy trg2src: 47.90 2021-07-14 12:49:51 - Save model to output/make-multilingual-en-zh_cn-zh_tw-2021-07-14_12-48-15 2021-07-14 12:50:23 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 150 steps: 2021-07-14 12:50:25 - Cosine-Similarity : Pearson: 0.2201 Spearman: 0.4078 2021-07-14 12:50:25 - Manhattan-Distance: Pearson: 0.4307 Spearman: 0.4912 2021-07-14 12:50:25 - Euclidean-Distance: Pearson: 0.2403 Spearman: 0.3601 2021-07-14 12:50:25 - Dot-Product-Similarity: Pearson: 0.0806 Spearman: 0.1509 2021-07-14 12:50:25 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 150 steps: 2021-07-14 12:50:27 - Accuracy src2trg: 40.60 2021-07-14 12:50:27 - Accuracy trg2src: 47.00 2021-07-14 12:50:53 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 200 steps: 2021-07-14 12:50:55 - Cosine-Similarity : Pearson: 0.3043 Spearman: 0.4286 2021-07-14 12:50:55 - Manhattan-Distance: Pearson: 0.4844 Spearman: 0.5083 2021-07-14 12:50:55 - Euclidean-Distance: Pearson: 0.2343 Spearman: 0.3075 2021-07-14 12:50:55 - Dot-Product-Similarity: Pearson: 0.0888 Spearman: 0.1209 2021-07-14 12:50:55 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 200 steps: 2021-07-14 12:50:57 - Accuracy src2trg: 39.00 2021-07-14 12:50:57 - Accuracy trg2src: 45.30 2021-07-14 12:51:24 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 250 steps: 2021-07-14 12:51:27 - Cosine-Similarity : Pearson: 0.2078 Spearman: 0.4172 2021-07-14 12:51:27 - Manhattan-Distance: Pearson: 0.4169 Spearman: 0.4681 2021-07-14 12:51:27 - Euclidean-Distance: Pearson: 0.2209 Spearman: 0.2994 2021-07-14 12:51:27 - Dot-Product-Similarity: Pearson: 0.0991 Spearman: 0.1113 2021-07-14 12:51:27 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 250 steps: 2021-07-14 12:51:28 - Accuracy src2trg: 35.70 2021-07-14 12:51:28 - Accuracy trg2src: 39.90 2021-07-14 12:51:56 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 300 steps: 2021-07-14 12:51:58 - Cosine-Similarity : Pearson: 0.2251 Spearman: 0.3829 2021-07-14 12:51:58 - Manhattan-Distance: Pearson: 0.3689 Spearman: 0.4266 2021-07-14 12:51:58 - Euclidean-Distance: Pearson: 0.2128 Spearman: 0.2842 2021-07-14 12:51:58 - Dot-Product-Similarity: Pearson: 0.0947 Spearman: 0.1040 2021-07-14 12:51:58 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 300 steps: 2021-07-14 12:52:00 - Accuracy src2trg: 32.40 2021-07-14 12:52:00 - Accuracy trg2src: 28.20 2021-07-14 12:52:27 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 350 steps: 2021-07-14 12:52:29 - Cosine-Similarity : Pearson: 0.1611 Spearman: 0.3470 2021-07-14 12:52:29 - Manhattan-Distance: Pearson: 0.2974 Spearman: 0.4102 2021-07-14 12:52:29 - Euclidean-Distance: Pearson: 0.1915 Spearman: 0.2501 2021-07-14 12:52:29 - Dot-Product-Similarity: Pearson: 0.1003 Spearman: 0.1044 2021-07-14 12:52:29 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 350 steps: 2021-07-14 12:52:31 - Accuracy src2trg: 20.50 2021-07-14 12:52:31 - Accuracy trg2src: 24.30 2021-07-14 12:52:59 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 400 steps: 2021-07-14 12:53:01 - Cosine-Similarity : Pearson: 0.1161 Spearman: 0.3565 2021-07-14 12:53:01 - Manhattan-Distance: Pearson: 0.3088 Spearman: 0.4184 2021-07-14 12:53:01 - Euclidean-Distance: Pearson: 0.1754 Spearman: 0.2324 2021-07-14 12:53:01 - Dot-Product-Similarity: Pearson: 0.0989 Spearman: 0.1024 2021-07-14 12:53:01 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 400 steps: 2021-07-14 12:53:03 - Accuracy src2trg: 18.90 2021-07-14 12:53:03 - Accuracy trg2src: 25.20 2021-07-14 12:53:30 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 450 steps: 2021-07-14 12:53:33 - Cosine-Similarity : Pearson: 0.1097 Spearman: 0.3585 2021-07-14 12:53:33 - Manhattan-Distance: Pearson: 0.3052 Spearman: 0.4132 2021-07-14 12:53:33 - Euclidean-Distance: Pearson: 0.1915 Spearman: 0.2409 2021-07-14 12:53:33 - Dot-Product-Similarity: Pearson: 0.0982 Spearman: 0.1022 2021-07-14 12:53:33 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 450 steps: 2021-07-14 12:53:34 - Accuracy src2trg: 13.80 2021-07-14 12:53:34 - Accuracy trg2src: 22.70 2021-07-14 12:54:01 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 500 steps: 2021-07-14 12:54:03 - Cosine-Similarity : Pearson: 0.1913 Spearman: 0.2505 2021-07-14 12:54:03 - Manhattan-Distance: Pearson: 0.2645 Spearman: 0.2693 2021-07-14 12:54:03 - Euclidean-Distance: Pearson: 0.2142 Spearman: 0.2391 2021-07-14 12:54:03 - Dot-Product-Similarity: Pearson: 0.1860 Spearman: 0.1852 2021-07-14 12:54:03 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 500 steps: 2021-07-14 12:54:05 - Accuracy src2trg: 4.20 2021-07-14 12:54:05 - Accuracy trg2src: 5.00 2021-07-14 12:54:30 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 550 steps: 2021-07-14 12:54:33 - Cosine-Similarity : Pearson: 0.0920 Spearman: 0.2471 2021-07-14 12:54:33 - Manhattan-Distance: Pearson: 0.2092 Spearman: 0.2441 2021-07-14 12:54:33 - Euclidean-Distance: Pearson: 0.1707 Spearman: 0.2083 2021-07-14 12:54:33 - Dot-Product-Similarity: Pearson: 0.1608 Spearman: 0.1507 2021-07-14 12:54:33 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 550 steps: 2021-07-14 12:54:34 - Accuracy src2trg: 8.60 2021-07-14 12:54:34 - Accuracy trg2src: 8.80 2021-07-14 12:55:00 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 600 steps: 2021-07-14 12:55:03 - Cosine-Similarity : Pearson: 0.0740 Spearman: 0.2366 2021-07-14 12:55:03 - Manhattan-Distance: Pearson: 0.1805 Spearman: 0.2219 2021-07-14 12:55:03 - Euclidean-Distance: Pearson: 0.1573 Spearman: 0.1914 2021-07-14 12:55:03 - Dot-Product-Similarity: Pearson: 0.1576 Spearman: 0.1474 2021-07-14 12:55:03 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 600 steps: 2021-07-14 12:55:04 - Accuracy src2trg: 8.70 2021-07-14 12:55:04 - Accuracy trg2src: 10.80 2021-07-14 12:55:31 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 650 steps: 2021-07-14 12:55:33 - Cosine-Similarity : Pearson: 0.0553 Spearman: 0.2483 2021-07-14 12:55:33 - Manhattan-Distance: Pearson: 0.1631 Spearman: 0.2338 2021-07-14 12:55:33 - Euclidean-Distance: Pearson: 0.1491 Spearman: 0.1845 2021-07-14 12:55:33 - Dot-Product-Similarity: Pearson: 0.1504 Spearman: 0.1374 2021-07-14 12:55:33 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 650 steps: 2021-07-14 12:55:35 - Accuracy src2trg: 9.70 2021-07-14 12:55:35 - Accuracy trg2src: 9.10 2021-07-14 12:56:02 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 700 steps: 2021-07-14 12:56:04 - Cosine-Similarity : Pearson: 0.0525 Spearman: 0.1912 2021-07-14 12:56:04 - Manhattan-Distance: Pearson: 0.1127 Spearman: 0.1620 2021-07-14 12:56:04 - Euclidean-Distance: Pearson: 0.1433 Spearman: 0.1756 2021-07-14 12:56:04 - Dot-Product-Similarity: Pearson: 0.1343 Spearman: 0.1226 2021-07-14 12:56:04 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 700 steps: 2021-07-14 12:56:06 - Accuracy src2trg: 6.30 2021-07-14 12:56:06 - Accuracy trg2src: 5.90 2021-07-14 12:56:32 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 750 steps: 2021-07-14 12:56:35 - Cosine-Similarity : Pearson: 0.0414 Spearman: 0.1971 2021-07-14 12:56:35 - Manhattan-Distance: Pearson: 0.1283 Spearman: 0.1865 2021-07-14 12:56:35 - Euclidean-Distance: Pearson: 0.1419 Spearman: 0.1620 2021-07-14 12:56:35 - Dot-Product-Similarity: Pearson: 0.1386 Spearman: 0.1243 2021-07-14 12:56:35 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 750 steps: 2021-07-14 12:56:36 - Accuracy src2trg: 6.40 2021-07-14 12:56:36 - Accuracy trg2src: 6.30 2021-07-14 12:57:02 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 800 steps: 2021-07-14 12:57:05 - Cosine-Similarity : Pearson: 0.0415 Spearman: 0.1840 2021-07-14 12:57:05 - Manhattan-Distance: Pearson: 0.1434 Spearman: 0.2044 2021-07-14 12:57:05 - Euclidean-Distance: Pearson: 0.1375 Spearman: 0.1334 2021-07-14 12:57:05 - Dot-Product-Similarity: Pearson: 0.1202 Spearman: 0.1056 2021-07-14 12:57:05 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 800 steps: 2021-07-14 12:57:06 - Accuracy src2trg: 7.10 2021-07-14 12:57:06 - Accuracy trg2src: 6.90 2021-07-14 12:57:33 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 850 steps: 2021-07-14 12:57:36 - Cosine-Similarity : Pearson: 0.0650 Spearman: 0.1913 2021-07-14 12:57:36 - Manhattan-Distance: Pearson: 0.1905 Spearman: 0.2185 2021-07-14 12:57:36 - Euclidean-Distance: Pearson: 0.1637 Spearman: 0.1580 2021-07-14 12:57:36 - Dot-Product-Similarity: Pearson: 0.1383 Spearman: 0.1169 2021-07-14 12:57:36 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 850 steps: 2021-07-14 12:57:37 - Accuracy src2trg: 4.80 2021-07-14 12:57:37 - Accuracy trg2src: 5.10 2021-07-14 12:58:04 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 900 steps: 2021-07-14 12:58:06 - Cosine-Similarity : Pearson: 0.0537 Spearman: 0.1571 2021-07-14 12:58:06 - Manhattan-Distance: Pearson: 0.1480 Spearman: 0.1842 2021-07-14 12:58:06 - Euclidean-Distance: Pearson: 0.1453 Spearman: 0.1404 2021-07-14 12:58:06 - Dot-Product-Similarity: Pearson: 0.1095 Spearman: 0.0961 2021-07-14 12:58:06 - Evaluating translation matching Accuracy on TranslationEvaluator dataset in epoch 0 after 900 steps: 2021-07-14 12:58:08 - Accuracy src2trg: 4.90 2021-07-14 12:58:08 - Accuracy trg2src: 4.70 2021-07-14 12:58:34 - EmbeddingSimilarityEvaluator: Evaluating the model on sts-dev dataset in epoch 0 after 950 steps: 2021-07-14 12:58:36 - Cosine-Similarity : Pearson: 0.0875 Spearman: 0.1750 2021-07-14 12:58:36 - Manhattan-Distance: Pearson: 0.1842 Spearman: 0.2187