loading model while finetuning

[ironllamagirl]

Hello! Thank you very much for the work being done on this package. It has helped me tremendously.

I am finetuning the model on custom data, using the triplet function. It has worked like a charm when the base model was 'bert-base-uncased'. I loaded it like this: word_embedding_model = models.BERT('bert-base-uncased')

The problem now is that I wanted to do the finetuning on a different 'base' model. I can't seem to include the already trained models provided such as distilbert-base-nli-stsb-mean-tokens or Roberta's. I am thinking using a more advanced model trained for the similarity task would yield better results but I can't seem to load it.

When I load the model using model = SentenceTransformer('model_name'), I get the following error later on during pooling: ''SentenceTransformer' object has no attribute 'get_word_embedding_dimension''

I would appreciate your thoughts on this. Thank you again

[nreimers]

Hi @ironllamagirl You need to differentiate between huggingface transformers models like bert-base-uncased, and SentenceTransformers models like distilbert-base-nli-stsb-mean-tokens. These models are already shipped with a pooling layer, i.e., adding it another time is not needed.

See this example how to load a SentenceTransformers model and to continue training on another dataset: https://github.com/UKPLab/sentence-transformers/blob/master/examples/training_transformers/training_stsbenchmark_continue_training.py

Basically you just load it: model = SentenceTransformer('model_name')

And then you call the fit method: model.fit(...)

[ironllamagirl]

Oh I see thank you very much!

[ironllamagirl]

Hello, I am reopening this since I am trying to fine-tune KoBert (Korean Bert) on my own data using the same code for finetuning provided in this package. Is this possible? since I believe it is not a huggingface model nor is it a sentenceTransformers model. Is it possible to load this model?

Here is the link to KoBert: https://github.com/SKTBrain/KoBERT#why

Thank you.

[nreimers]

Hi @ironllamagirl I think you can find the huggingface KoBERT Model here: https://huggingface.co/monologg/kobert

Just specify 'monologg/kobert' as your model in the training scripts here and it should work.

Best Nils

[ironllamagirl]

Thank you! I am using these lines of code to load the model from huggingface:

word_embedding_model = models.Transformer('monologg/kobert')

Apply mean pooling to get one fixed sized sentence vector

pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(), pooling_mode_mean_tokens=True, pooling_mode_cls_token=False, pooling_mode_max_tokens=False)

model = SentenceTransformer(modules=[word_embedding_model, pooling_model])

But I am getting the following error in the first line. AttributeError: module 'sentence_transformers.models' has no attribute 'Transformer'

[nreimers]

Hi @ironllamagirl Have you tried to update to the most recent version?

[ironllamagirl]

Updating to the most recent version fixed the issue. Cheers!