adapt number of output files based on document type

rom1504 / cc2dataset

Easily convert common crawl to a dataset of caption and document. Image/text Audio/text Video/text, ...

MIT License

307 stars 23 forks source link

Open rom1504 opened 1 year ago

rom1504 commented 1 year ago

shard / 100 for image only shard / 200 for image lot less for other types