Open srcarroll opened 3 years ago
You misunderstand my question. I'm asking why the script generates an empty file. It turns out that it's because the get-data-wiki script is completely riddled with problems and doesn't work as intended. I've already moved on
On Mon, Oct 26, 2020 at 6:25 AM arjunkoneru notifications@github.com wrote:
en.train is your training data for english. how can you learn the codes if it is empty?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/facebookresearch/XLM/issues/319#issuecomment-716543764, or unsubscribe https://github.com/notifications/unsubscribe-auth/AL7CPJYTAD55AXTJMZOG3MTSMV2FLANCNFSM4SRATXAA .
Maybe you can use this Flores mono data and remove the part where it downloads nepali and sinhala
I met the same problem, and it is probably due to the project named wikiExtractor ,when the script runs to clean and tokenize, there is an error about losing defination. I finally solve it by following https://github.com/attardi/wikiextractor/issues/222, and don't forget modify the corresponding command in the get-data-wiki.sh
I'm trying to go through the steps in your readme and getting stuck on the fastBPE part of the "Preparing the data" section. When I try to run
$FASTBPE learnbpe 30000 data/wiki/txt/en.train > $OUTPATH/codes
I get the following output
Loading vocabulary from data/wiki/txt/en.train ... Read 0 words (0 unique) from text file. Segmentation fault (core dumped)
checking en.train shows that it's empty. Any idea of the issue causing this?