allenai / bilm-tf

Tensorflow implementation of contextualized word representations from bi-directional language models
Apache License 2.0
1.62k stars 452 forks source link

Unable to get bilm embeddings from my pretrained model #192

Closed smeaktrobush closed 5 years ago

smeaktrobush commented 5 years ago

I cannot get embeddings from my pretrained model. I have tried using dump_bilm_embeddings method but it still doesn't work. Every time I run the method, the kernel just straight up crashes. The best I could get from the error is Error in `/home/punn_pnt/anaconda3/envs/allennlp/bin/python': free(): invalid next size (fast): 0x00007fa13400d040 '

This is my code `options_file = "/home/punn_pnt/senior_proj/bilm-tf/checkpoint/options.json" weight_file = "/home/punn_pnt/senior_proj/bilm-tf/checkpoint/weights.hdf5" vocab_file = "/home/punn_pnt/senior_proj/bilm-tf/vocab.txt"

batcher = Batcher(lm_vocab_file=vocab_file,max_token_length=50)

sentence = ['some','sentence', .....]

ids_placeholder = tf.placeholder(tf.int32,shape=(None,None,50)) elmo=BidirectionalLanguageModel(options_file=options_file,weight_file=weight_file) ops=elmo(ids_placeholder)

config = tf.ConfigProto(allow_soft_placement=True) with tf.Session(config=config) as sess: sess.run(tf.global_variables_initializer()) ids = batcher.batch_sentences([sentence]) embeddings = sess.run(ops['lm_embeddings'], feed_dict={ids_placeholder: ids}) `

I am on Ubuntu 16.04 LTS, Python 3.6.8 Anaconda, allennlp 0.8.3

Thank you