Closed Jason-kid closed 2 years ago
Hi,
The MLM training is know for causing "representation degeneration", which means the embeddings of different words/sentences become similar.
thank for you kind apply ! well, i have an another question. i have a dataset consist of a line of two sentence, which not labels whether they are similar or not . for the task above , i try to train the model in the unsupervised way, where i group the sentA and sentB into one sentence for the unsupervised training. Do you have some kind advice ? thanks a lot .
when i add do_mlm flag, cosine similarity will become larger even two sentences are irrelevant?