[PR] Add char-speed distribution calculations - Githubissues

HarikalarKutusu / cv-tbox-dataset-compiler

GNU Affero General Public License v3.0

0 stars 0 forks source link

[PR] Add char-speed distribution calculations #38

Closed HarikalarKutusu closed 1 month ago

HarikalarKutusu commented 1 month ago

Added Features

feat: Add char-speed distribution calculations
- Make use of alternate binning for logogram languages (at threshold 300 msec/char speed)
- [OK] Char-speed bins
- [OK] Sentence length alternate bins for char-speed tab
feat: Add validated duration to support_matrix for showing colored info on Analyzer. It also shows a tooltip
feat: Make use of alternate sentence length bins for text-corpus tab

Programming Changes

feat: Add rudimentary multiprocessing schedular
chore: convert some list/array data (back) into strings for maximum compatibility / reduced data size
chore: Divide large final_compile.py file
chore: Remove temporary save for HDD performance

Other

We now limit the grapheme/phoneme lists to most common 100, as for logographic languages the list can become very large.