search

hotpotqa / hotpot

Apache License 2.0
445 stars 75 forks source link

UnboundLocalError: local variable 'word_counter' referenced before assignment：during preprocessing #35

Closed qizitong closed 4 years ago

qizitong commented 4 years ago

when i want to give below command as per your manual at here.. Three commands have different errors python main.py --mode prepro --data_file hotpot_train_v1.1.json --para_limit 2250 --data_split train （ [Parallel(n_jobs=12)]: Done 35960 tasks | elapsed: 2.4min [Parallel(n_jobs=12)]: Done 36784 tasks | elapsed: 2.5min [Parallel(n_jobs=12)]: Done 37624 tasks | elapsed: 2.5min [Parallel(n_jobs=12)]: Done 38464 tasks | elapsed: 2.5min [Parallel(n_jobs=12)]: Done 39320 tasks | elapsed: 2.6min [Parallel(n_jobs=12)]: Done 40176 tasks | elapsed: 2.6min [Parallel(n_jobs=12)]: Done 41048 tasks | elapsed: 2.7min [Parallel(n_jobs=12)]: Done 41920 tasks | elapsed: 2.7min [Parallel(n_jobs=12)]: Done 42808 tasks | elapsed: 2.8min 已杀死 ）

python main.py --mode prepro --data_file hotpot_dev_distractor_v1.json --para_limit 2250 --data_split dev （ [Parallel(n_jobs=12)]: Done 7405 out of 7405 | elapsed: 30.2s finished 7405 questions in total Traceback (most recent call last): File "main.py", line 86, in prepro(config) File "/home/caoxing/project/hotpot-master/prepro.py", line 324, in prepro word_emb_mat, word2idx_dict, idx2word_dict = get_embedding(word_counter, "word", emb_file=config.glove_word_file, UnboundLocalError: local variable 'word_counter' referenced before assignment ）

python main.py --mode prepro --data_file hotpot_dev_fullwiki_v1.json --data_split dev --fullwiki --para_limit 2250

（[Parallel(n_jobs=12)]: Done 7405 out of 7405 | elapsed: 30.3s finished 7405 questions in total Traceback (most recent call last): File "main.py", line 86, in prepro(config) File "/home/caoxing/project/hotpot-master/prepro.py", line 324, in prepro word_emb_mat, word2idx_dict, idx2word_dict = get_embedding(word_counter, "word", emb_file=config.glove_word_file, UnboundLocalError: local variable 'word_counter' referenced before assignment ） I would be very appreciated if you can help me:)

qipeng commented 4 years ago

You're seeing the UnboundLocalError because you didn't finish processing the training set before starting to process the test set. Note that the process processing the training set failed. Try using fewer jobs to reduce RAM usage?

qizitong commented 4 years ago

Thank you for your reply. The problem has been solved.

在 2020年5月7日，上午4:32，Peng Qi notifications@github.com 写道：

You're seeing the UnboundLocalError because you didn't finish processing the training set before starting to process the test set. Note that the process processing the training set failed. Try using fewer jobs to reduce RAM usage?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hotpotqa/hotpot/issues/35#issuecomment-624875006, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHFLVG75XV32ZRALBPQ35ITRQHCNZANCNFSM4M2MXZZA.