UnitY2/ProsodyUnitY2 model training - Githubissues

facebookresearch / seamless_communication

Foundational Models for State-of-the-Art Speech and Text Translation

Other

10.94k stars 1.06k forks source link

UnitY2/ProsodyUnitY2 model training #379

Closed yilinyang7 closed 8 months ago

yilinyang7 commented 8 months ago

This is an WIP version of UnitY2/ProsodyUnitY2 training codebase. It includes (I'll write more details in a seperate doc):

loss computation (at unity/loader.py)
dataset class written 100% with fairseq2.data API (unity2/dataset.py)
trainer class & pipeline (unity2/trainer.py)
implemented trainable UnitYAlignerModel, UnitYNART2Umodel, UnitYModel, etc..
yaml-configurable multi-task learning model (multi_task_configs.py)
cli-configurable multi-task learning objective (unity2/trainer.py)

Still WIP part:

multi-node job launcher
UnitYMetricBag class (low-pri)

There're also numerious backward-compatible changes.