Closed boss-chanon closed 1 year ago
don't save tokenized to chunk
Close #
@boss-chanon please resolved conflict
plz review again because i convert to chunk because large dataset will out of memory if don't chunk but i fix load tokenized dataset to can load chunk data
Why this PR
don't save tokenized to chunk
Changes
Related Issues
Close #
Checklist