Train on mid-verse USFM - and generate for NLLB fine tuned models

sillsdev / silnlp

A set of pipelines for performing experiments on various NLP tasks with a focus on resource-poor/minority languages.

Other

30 stars 4 forks source link

Train on mid-verse USFM - and generate for NLLB fine tuned models #181

Open johnml1135 opened 11 months ago

johnml1135 commented 11 months ago

Ideas to enable this could be:

Train a model with the mid-verse USFM tokens (or replacement tokens) that it learns to place properly.
Reinsert the USFM after generation of the draft by using the attention of the models, summed up for each subsection (between intra-verse USFM markers), train a Spacy model to re-inject the USFM markers. @ddaspit, what do you think? It shouldn't be that hard to try out...

johnml1135 commented 11 months ago

This is for researching whether NLLB can do it at all - if it can, we have to determine the best delivery method. A paratext plugin would likely work the best as Scripture Forge does not render and can break on mid-verse USFM.