Closed codertimo closed 5 years ago
Hi, Junseong. I think you should sampling per every batch to prevent overfitting. And I have a question. how do you tokenize your dataset? Their vocab size is only 30000(subword level) because of their internal source (WPM).
Thank you for asking @DSKSD , good to see you on opensource community
Hmm I worried about overfitting too if we using the generated(random sampled) corpus, every epoch. I'm not sure how the google guys did it, maybe I can send the email to author?
The paper refer that they used the wordpiece model which google ai developed (it might be different with sentence-piece). So input corpus should be tokenized. In my code, we just only tokenize with space character. (add+ I used enterprise internal tokenization model for korean corpus)
Hi, Junseong, As far as I know, wordpiece can be obtained easily with learn_bpe and apply_bpe in machine translation open source code (OpenNMT and tensor2tensor). You can try it! However, the effect of BPE in Chinese is not as good as that in English. I believe similar phenomenon will be oberseved in Korean as well.
@DSKSD 0.0.1a3 version updated for random sampling per batch
Is is okay to use random sampled data which is saved before the training? I mean, should it have to be changed every epoch?