Could not broadcast input array from shape (299) into shape (300)

[bakszero]

I'm using Glove pre-trained vectors (840B, 300D, Common Crawl). When I try to train the system, I get the following error:

Configurations:
Namespace(MP_dim=20, NER_dim=20, POS_dim=20, aggregation_layer_num=1, aggregation_lstm_dim=100, batch_size=60, char_emb_dim=20, char_lstm_dim=100, context_layer_num=1, context_lstm_dim=100, dev_path='./dev.tsv', dropout_rate=0.1, fix_word_vec=True, highway_layer_num=1, lambda_l2=0.0, learning_rate=0.001, lex_decompsition_dim=-1, max_char_per_word=10, max_epochs=10, max_sent_length=100, model_dir='models', optimize_type='adam', suffix='sample', test_path='./test.tsv', train_path='./train.tsv', with_NER=False, with_POS=False, with_aggregation_highway=False, with_filter_layer=False, with_highway=False, with_lex_decomposition=False, with_match_highway=False, wo_attentive_match=False, wo_char=False, wo_full_match=False, wo_left_match=False, wo_max_attentive_match=False, wo_maxpool_match=False, wo_right_match=False, word_level_MP_dim=-1, word_vec_path='./glove.840B.300d.txt')
Traceback (most recent call last):
  File "src/SentenceMatchTrainer.py", line 391, in <module>
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
  File "/home/syed.b/stanford/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 43, in run
    sys.exit(main(sys.argv[:1] + flags_passthrough))
  File "src/SentenceMatchTrainer.py", line 128, in main
    word_vocab = Vocab(word_vec_path, fileformat='txt3')
  File "/home/syed.b/Research/BiMPM/src/vocab_utils.py", line 18, in __init__
    self.fromText_format3(vec_path,voc=voc)
  File "/home/syed.b/Research/BiMPM/src/vocab_utils.py", line 143, in fromText_format3
    self.word_vecs[cur_index] = word_vecs[cur_index]
ValueError: could not broadcast input array from shape (299) into shape (300)

The format of the Glove txt file I'm using is: < word > < 1st dim.value > < 2nd dim. value > ..... < 300th dim. value > (without the relational signs of course)

[uduse]

I had a similar problem earlier and that was caused by an incomplete file.

[theSage21]

@uduse what do you mean by an incomplete file? I've just downloaded the glove zip and unzipped it. I've also verified that all vectors have 300 items in that file. What else could be missing?

[uduse]

@theSage21 somehow my file is truncated so some of the vectors aren't complete. So you're having the same problem?

[theSage21]

Here's what i did to verify all vectors are 300 dim.

with open('glove.txt', 'r') as fl:
    for line in fl.readlines():
        w, v = line.split(' ', 1)
        assert (v.strip().split(' ')) == 300

This ran without errors on my machine so I assume that all vectors have 300 dims. Despite this I have the same traceback.

I downloaded glove from https://nlp.stanford.edu/data/glove.840B.300d.zip

[uduse]

@theSage21 try w, v = line.strip().split(' ', 1) I doubt there's a weird char that got stripped away

[theSage21]

It seems that one of the vectors in the Glove file is for a space (line 142319) which gets removed when we strip() causing a total of 300 items in the parts list instead of 301. I've submitted a fix for this in https://github.com/zhiguowang/BiMPM/pull/26

@zhiguowang does this seem ok?

[uduse]

@theSage21 the project isn't maintained. I wouldn't expect a merge but it's nice to have for anyone who has the same problem.