[FR] Add one more splitting algorithm for testing

HarikalarKutusu / cv-tbox-split-maker

Checks diversity in Mozilla Common Voice default or alternative splits for multiple versions and languages

Mozilla Public License 2.0

1 stars 0 forks source link

Closed HarikalarKutusu closed 9 months ago

HarikalarKutusu commented 1 year ago

nv: seNtences-first w. unique Voices

Sort by unique sentences recording count, distribute to test, dev, rest to train, 80-10-10%
One voice only in one split
Ensure voice diversity 25-25-50% as in v1 algorithm

Most probably there will be cases that this algorithm will fail to use the whole dataset as it tries to enforce both sentence and voice diversity.

HarikalarKutusu commented 9 months ago

We added two other (vw & vx), but will not ad the one above.