Fix preprocess data file - Githubissues

Oneflow-Inc / libai

LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training

https://libai.readthedocs.io

Apache License 2.0

391 stars 55 forks source link

Fix preprocess data file #431

Closed xiezipeng-ML closed 1 year ago

xiezipeng-ML commented 1 year ago

修复一下tools/preprocess_data.py中缺失设置merge_file的问题
refine nltk.download("punkt", quiet=True)下载无提示，用户不知道卡在哪的问题，并提供手动下载地址