issues
search
OpenThaiGPT
/
openthaigpt-pretraining
Apache License 2.0
21
stars
10
forks
source link
docs(data): add info on deduplication
#303
Closed
new5558
closed
12 months ago
new5558
commented
12 months ago
Why this PR
To have documentation on deduplication code
Changes
Add readme for deduplication script
Related Issues
Checklist
[ ] PR should be in the
Naming convention
[ ] Assign yourself in to Assigneees
[ ] Tag related issues
[ ] Constants name should be ALL_CAPITAL, function name should be snake_case, and class name should be CamelCase
[ ] complex function/algorithm should have
Docstring
[ ] 1 PR should not have more than 200 lines changes (Exception for test files). If more than that please open multiple PRs
[ ] At least PR reviewer must come from the task's team (model, eval, data)
Why this PR
To have documentation on deduplication code
Changes
Related Issues
Checklist