Closed emielke12 closed 5 years ago
In principle, you don't have to do any special formatting. However, in the original paper, we used some pre-processing to reduce the size of the vocabulary and improve tokenization. This is beneficial if the text you are dealing with has to do with materials science/chemistry. You can use the process method here, then join back the tokens and dump it to the text corpus file. Let me know if this answers your question and I will close the issue.
Yes this answers my question. Thanks!
Is there any special text formatting that needs to be done to abstracts before training? I noticed the corpus example has
%
and<nUm>
in various places. Just wondering if formatting matters at all, or if you can dump the plain text from abstracts into a corpus file.