OpenThaiGPT / openthaigpt-pretraining

Apache License 2.0
21 stars 10 forks source link

Add new line token in tokenizer #282

Open boat1603 opened 1 year ago

boat1603 commented 1 year ago

Why this PR

Why we need this PR? This PR is about add \n token in the tokenizer trainer.

Changes

Related Issues

Close #

Checklist