Falling back to regular transformers behaviour in v0.3.5

[ogencoglu]

Checking the latest (v0.3.5) release notes:

If a sentence-transformer model is not found, it will fall back to huggingface transformers repository and create it with mean pooling.

What is the logic behind this? This sounds like an unnecessary babysitting that can cause severe academic reproducibility issues, especially for programming beginners. One may be new to programming, python, etc. and may have an impression that they are using a sentence transformer model while in fact they are not. Several academic publications have been invalidated due to similar kind of behavior in other libraries and fields.

[nreimers]

The flow is the following.

When you call:

model = SentenceTransformer('model_name')

Then it will perform the following steps: 1) Is there a folder with the name 'model_name'? Yes, then try to load the model from there. 2) If there is no folder, check if there is a pre-trained model at https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/v0.2/. If no, output a warning and go to the next step 3) If there is no pre-trained model on our server, check if the model is at https://huggingface.co/models. If yes, create an AutoModel and a mean pooling layer with that model.

Step 3) was added in the latest version.

In the previous version:

model = SentenceTransformer('bert-base-uncased')

was leading to an error, as there is no bert-base-uncased model on our server.

In the new version, it will load 'bert-base-uncased' with mean pooling.

The logic behind this is:

• There are an increasing number of suitable sentence embedding models on huggingface.co/models. As long as they are using mean pooling, they can be loaded easily.
• It will become easier and less confusing to create a sentence transformer model from pre-trained models. Previously SentenceTransformer('bert-base-uncased') was throwing an exception and people wondered why. Now, it will load bert-base-uncased with mean pooling. They can then use SentenceTransformer('bert-base-uncased') and fine-tune it on their data.

What I could add, to avoid the points you mention, are the following:

• Making the warning more clear, that they load a model from huggingface/models which is maybe not fine-tuned for sentence embeddings. They should check out the models card on huggingface/models what the purpose of the model is.
• Adding a prompt to to require user confirmation

[ogencoglu]

Thanks for the swift reply. Solutions you provided sound relevant indeed.