search

rom1504 / cc2dataset

Easily convert common crawl to a dataset of caption and document. Image/text Audio/text Video/text, ...
MIT License
307 stars 23 forks source link

shuffle #16

Closed rom1504 closed 1 year ago

rom1504 commented 1 year ago

.sort(rand())

testing perf

rom1504 commented 1 year ago

worked in 30min for 20B items, including

rom1504 commented 1 year ago

done