kyzhouhzau / BERT-NER

Use Google's BERT for named entity recognition (CoNLL-2003 as the dataset).
MIT License
1.24k stars 335 forks source link

Problems: runs very slowly when converting single example to feature #29

Closed jiangpinglei closed 5 years ago

jiangpinglei commented 5 years ago

I found it cost too much time When running the convert_single_example function.

time0 = time.time() feature = convert_single_example(ex_index, example, label_list, max_seq_length, tokenizer,mode) time1 = time.time() print("time cost:", time1-time0)

the cost is up to a few seconds! time cost: 4.020495414733887

In convert_single_example function we can fix it by add the following code : if not os.path.exists('./output/label2id.pkl'):

in front of

with open('./output/label2id.pkl','wb') as w: pickle.dump(label_map, w)