Fine-Tuning STS and semantic search multilingual HF transformer

[Matthieu-Tinycoaching]

Hi,

For the need of exporting transformer model to ONNX format for inference, I use a multilingual sentence-transformer model based on the HF transformers library (separate tokenizer and model + mean pooling layer) for semantic text similarity and semantic search.

Is it possible to fine-tune this model with # https://github.com/UKPLab/sentence-transformers/tree/master/examples/training/sts/training_stsbenchmark_continue_training.py despite not using the sentence-transformer library directly?

If not is there a workaround to enable to do this? (like fine-tune sentence-transformer model then split tokenizer/model before ONNX export)

Thanks!

[nreimers]

I would first fine tune the model an then export to onnx

[Matthieu-Tinycoaching]

Thanks @nreimers I would have done sequentially the steps you said.

The problem is that fine-tuning sentence-transformer model seems to be clear following the steps at https://github.com/UKPLab/sentence-transformers/tree/master/examples/training/sts/training_stsbenchmark_continue_training.py

But for exporting model to ONNX format I use the following command:

    convert_graph_to_onnx.convert(
        framework="pt",
        model=model_name,
        output=model_pth,
        opset=12,
        tokenizer="xlm-roberta-base",
        use_external_format= False,
        pipeline_name= pipeline_name,
    )

which is based on HF implementation of sentence-transformer separating tokenizer from model, itself not having mean pooling step. How could I convert then fine-tuned sentence-transformer model into tokenizer and model as expected by HF ?

Thanks!

[nreimers]

When the model is saved, you get a regular transformers and tokenizers models which you can then convert

[Matthieu-Tinycoaching]

That sounds great @nreimers! So, regular transformer model without mean pooling?

Could you give me an idea of how many minimum training examples would be needed to gain from fine-tuning pre-trained multilingual sentence-transformer for STS and semantic search?

[nreimers]

Correct.

Number of examples depends on how complex is your domain. With a simple and narrow domain 1k examples are helpful.

With a broad domain that spans over basically all topics (physics, math, sports, gaming, dating, programming,...) you need a lot more examples. If you don't have examples for eg math, the model will not work that well for math queries.