Mix multiple sources and one target

[johnml1135]

Spin off of https://github.com/sillsdev/serval/issues/266 for Serval implemenatation.

Mixing multiple sources from different NLLB-200 languages has shown to make a big bump especially if the backtranslation language is different than the source text (say English backtranslation with Spanish source text). Including target sentences in multiple times makes the behavior worse (memorizing the target sentences), but interweaving half the target with 2 different sources works pretty well. How can this be implemented:

Proposal:

• If an engine has two corpora: (1) Source123:TargetABC and (2) Source456:TargetABC
• A new optional parameter is added to corpora called mixingRatio which defaults to 1.0 - but can accept any valid float number including 1e100, etc..
• If a textId appears multiple times in corpora, the target is mixed with the different sources according to the weight:
  • 1.0-1.0 -> 50% and 50%: this is the default
  • 1.0-1.0-1.0 -> 33%, 33% and 33%: can combine more than 2 source languages
  • 1.0-2.0 -> 33.3% and 66.7%: by setting one corpora to a higher mixingRatio (such as 2.0), a source can take be aligned with more of the target sentences
• Mixing is done with repeatable, random seeds, not just ABABABABABABAB.
• If a source only covers a subset of the target, the mixing is done for the sources that are available on a segment by segment basis.
  • I.e., if there is only a backtranslation for Luke, but the source also has Mark and Matthew, Luke will be mixed as per the mixingRatio specified and Mark and Matthew will be exclusively from the other source.
  • To specify an ordering of sources, using exponents should be straightforward: 1e0, 1e10, 1e20, 1e30, each multiplying by a factor of 10,000,000,000, setting a clear priority for exclusive use by sources if they are available.

[johnml1135]

So, 50% random mix, if one source has no text, use the other source. Only 2 sources needed. Follow SILNLP implementation.

[johnml1135]

Proposal: for implementation, when posting the single corpora:

• For text files, make two source texts have the same textId - and then Machine will mix it
• For Paratext files, just put two zipped Paratext sources and the mixing will happen
• We will mix however many sources there are
• If there are multiple targets we error out. (UnsupportedException)

[johnml1135]

@ddaspit - here is a proposal of the changes:

• In CorpusService, look for the text and parallel matching conditions, and if there are multiple for the source, combine them (2 or more) into a MixedTextFileText or a MixedScriptureTextCorpus (may need to be MixedZippedScriptureTextCorpus...), both inherited from TextFileText and ScriptureTextCorpus.
• A MixedText class is made that the others inherit from that handles the radnomization
• The Mixed Corpora handle the randomizing when initially being created - randomizing whenever they have matching id's, but are treated as normal text or Paratext files otherwise

[ddaspit]

We need API documentation for this feature.