directly process lines while reading corpus

Determined22 / zh-NER-TF

A very simple BiLSTM-CRF model for Chinese Named Entity Recognition 中文命名实体识别 (TensorFlow)

2.33k stars 936 forks source link

directly process lines while reading corpus #93

Closed dedztbh closed 2 years ago

dedztbh commented 5 years ago

As issue #92 mentioned, reading all lines into the memory beforehand could cause potential OOM for large corpus. It's better to directly process each line while reading it of the corpus.

bobobe commented 2 years ago

Thanks，I'll reply you soon