General Preprocessing Pipeline

[kojimano]

• Integrate preprocessing script into the Megatron script and preprocess into mmap format (HF blog, script)
• Design options
  • tokenizer (Rinna)
  • 正規化
  • 重複除去