Conversion to fastText format

DavidNemeskey / cc_corpus

Tools for compiling corpora from Common Crawl

GNU Lesser General Public License v3.0

12 stars 1 forks source link

Open DavidNemeskey opened 3 years ago

DavidNemeskey commented 3 years ago

Add a fastText conversion option to convert_tsv.py. In the fastText input format,

each document should be written on a separate line (to avoid "cross pollination")
headers should be excluded