Is it possible to fine-tune the multilingual model?

[zjmtlab]

Thanks for your great work! It really great!

Maybe a stupid question, we want to use the multilingual model you provided to do similar sentence search on Chinese data, and want to know can we use our own data(Chinese data) to fine-tune the multingual model?

I have tried it but the fine-tuned model get reduced performance. So I want to know what should I do to use our own data to fine-tune it? Maybe the multilingual model fine-tune is different with the training_stsbenchmark_continue_training.py?

Thanks very much for any suggestion!

[nreimers]

Hi @zjmtlab I currently write the paper on how to train multi-lingual models. The paper and the respective code will be published soon.

You will then have a detailed guide + the needed code to train your models. I will let you know as soon as the paper is published.

Best Nils Reimers

[zjmtlab]

Thanks very much for your response!

And I want add a little explanation that we do't need cross-lingual, we use multi-lingual just because lack of chinese model, and we think this multi-lingual model is better than the model which we need train from scratch.

Let's wait your great paper! Thanks again!

Best Regards, Jian

[nreimers]

If you have already some suitable chinese data (like NLI oder STS), you can already train a Chinese model with the current version. Just plug your data into for example the training_nli example and use a multilingual BERT model (or XLM-RoBERTa).

Getting a cross-lingual model with aligned vector spaces (so that an English and Chinese sentence with similar meaning is mapped to the same location) is a bit more complex. But more details soon.

[zjmtlab]

ok, I will try it, thanks very much!

[Neuronys]

Hi Nils When do you plan to release your new paper and the code update ? Thanks for this current version anyway 😊

[nreimers]

Hi @Neuronys I just published the code to make models multi-lingual: https://github.com/UKPLab/sentence-transformers/blob/master/examples/training_multilingual.py

In that example we extend the English 'bert-base-nli-stsb-mean-tokens' model to German.

More details will follow next week. Hopefully the paper will also be ready by next week.

Best Nils

[Neuronys]

Thanks Nils. Really impatient to read your paper. Cheers

[Neuronys]

Hi @nreimers Have you released your paper ? Cheers Philippe

[nreimers]

Hi @Neuronys If you send me an email (info@nils-reimers.de) I can forward the paper to you. It is not yet available online.

Best Nils

[kasramsh]

Hi @Neuronys If you send me an email (info@nils-reimers.de) I can forward the paper to you. It is not yet available online.

Best Nils

Dear Nils,

Could you please forward the paper to me please (kasramsh@gmail.com)? Thanks in advance!

Best, Kasra