Closed deepankar27 closed 3 years ago
I was wondering the same thing. Would the authors consider publishing the trained model parameters, in addition to the word vectors, so we could do transfer learning on small data sets?
What model parameters are you referring to? The word vectors are the only output of the model.
As to whether you can use this for transfer learning, that's possible in the same way it's possible with any word vectors, though unless your data is similar to Reddit in some way you might be better off training from scratch.
Thank you @polm. I was referring to the neural net architecture and learned weights, that essentially takes a token and spits out the word vector. I was under the impression that we need those to do transfer learning. If I'm mistaken, I'd appreciate it if you could point me to the documentation on how to do transfer learning for s2v!
I'm considering transfer learning because we don't have 1 billion words. Our sample size is much smaller.
The way sense2vec works is it preprocesses text to handle both tokenization and labelling of tokens. You can see that in the preprocessing script. Here's example output of the process:
Rats|NOUN ,|PUNCT mould|NOUN and|CCONJ broken_furniture|NOUN :|PUNCT
the|DET scandal|NOUN of|ADP the|DET UK|GPE 's|PART refugee_housing|NOUN
This output is then fed to a typical word vector training algorithm, like GloVe or fasttext. There is no sense2vec-specific model.
I'm considering transfer learning because we don't have 1 billion words. Our sample size is much smaller.
You do not need one billion words to train a useful model. I would recommend at least trying out training vanilla word vectors and sense2vec on your data and seeing how well they work.
If you want to do transfer learning on the pretrained models anyway, there's no special process for that, it would work like any other word vectors, the main difficulty is that due to the preprocessing it won't be compatible with normal tokenizers.
Can we use this existing model for transfer learning on closed domain dataset?