Mix training sources with different languages

sillsdev / serval

A REST API for natural language processing services

MIT License

4 stars 0 forks source link

Mix training sources with different languages #354

Open johnml1135 opened 3 months ago

johnml1135 commented 3 months ago

This may specifically allow mixing sources with multiple languages - such as back translations in multiple languages. Mixing is important here, where there may be a back translation that is a different language than the desired source translation.

Possible implementation 1:

If no corpora source or target language is specified, use the Paratext language code in Settings.xml Possible implementation 2:
Add a language code to each text file as an override to the corpora language

ddaspit commented 3 months ago

Can we reframe this in terms of the problem instead of the solution?

johnml1135 commented 2 weeks ago

Let's say a team has sources in multiple languages (different parts of the back translation or reference bibles) for different portions of Scripture. Let's say they found the magical combination using the onboarding app/clowder and now they want to apply it to SF. How would they do that (both in the SF context and then in how SF would talk with Serval).

This appears to be a bit of an edge case - how much would we want to prioritize this?