Closed soldni closed 5 months ago
added flag --sample_ring_prop to dolma tokens (off by default). When set to true, sampling from files in the ring is done proportionally to their size so that smaller files are not disproportionately represented in the beginning of each file.
--sample_ring_prop
dolma tokens
added flag
--sample_ring_prop
todolma tokens
(off by default). When set to true, sampling from files in the ring is done proportionally to their size so that smaller files are not disproportionately represented in the beginning of each file.