search

TalSchuster / CrossLingualContextualEmb

Cross-Lingual Alignment of Contextual Word Embeddings
MIT License
98 stars 9 forks source link

How to you generate pre-trained embedding? #5

Closed Liangtaiwan closed 5 years ago

Liangtaiwan commented 5 years ago

@TalSchuster I'm wondering how do you generate the pretrained model? I have the following questions, Were you using https://github.com/allenai/bilm-tf with the default argument? How long did you train for each model? And how many gpu and what kind of gpu you used?

TalSchuster commented 5 years ago

Hi @Liangtaiwan. Thank you for showing interest. I'm trying to find time to train models for additional languages. If other people have trained such models, I welcome all to contribute to the repo!

About your questions: Yes, I've used the default parameters from the ELMo training code (10 epochs). The training time is dependent on the amount of unlabeled data. For English, it took about a week or so. For other languages, it wasn't that bad and if I remember correct was about a day of training with 3 gpus. I think I used 1080 for most of the models but I used also some other machines with newer gpus. Sorry for the not precise answers but I hope that you will find them helpful.

Liangtaiwan commented 5 years ago

@TalSchuster Thanks for your reply. My research is focus on multi-lingual embedding and zero-shot on low resources language, too. It's nice to have the conversation with you, thanks again for your reply.

azamatolegen commented 4 years ago

I am new to this area. I have a question, you have pre-trained the ELMO model, does it mean that we can "extract" word from that model? What I mean is we took English weights.hdf5 file and we can extract a contextual embedding of word 'bank' from it? Is that the way how it works? Or we still need to feed some data with sentences containing the word 'bank' to the model? (at least some small size data)

TalSchuster commented 4 years ago

Hi @azamatolegen,

ELMO can output contextualized word embeddings. You can either feed it with a single-word sentence 'bank' and get a single vector, or feed it with different sentences containing that word to retrieve different contextual embeddings.

azamatolegen commented 4 years ago

Hi @azamatolegen,

ELMO can output contextualized word embeddings. You can either feed it with a single-word sentence 'bank' and get a single vector, or feed it with different sentences containing that word to retrieve different contextual embeddings.

Thank you, sir.