on TSDAE

[lukemao]

Hi,

Thanks for publishing and sharing the TSDAE approach. I am reading through the paper. I have one question.

In Section 7.4 of the paper, it recommends using TSDAE as the pretrained approach before in-domain fine-tune.

Can we use TSDAE as the fine-tune training method after doing a Out-of-the-box supervised pre-trained models like USE-large or SBERT-base-NLI.

e.g.:

model_name = 'sentence-transformers/bert-large-nli-stsb-mean-tokens' # as the pretrained
word_embedding_model = models.Transformer(model_name)
pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(), 'cls')
model = SentenceTransformer(modules=[word_embedding_model, pooling_model])

# then do the training

Kindly regards, Lu.

[nreimers]

Sadly this does not work well. You first need to run TSDAE, then do training with your labeled data.

[lukemao]

thanks Nils!

[bdferris642]

Hey @nreimers!

When performing fine-tuning on top of a TSDAE-pretrained model, is it recommended to use the same pooling method as was used during TSDAE for fine-tuning and/or at inference time? E.g. if TSDAE was performed with pooling_model = models.Pooling(word_embedding_model.get_word_embedding_dimension(), 'cls') , should 'cls' pooling be used throughout? Thanks!

[nreimers]

@bdferris642 Yes, it makes sense to use the same pooling

[bdferris642]

Thank you!