Feature: Localization Processors

[sttz]

This is a slightly exotic feature request. I won't mind if you close this request but I still wanted to bring it up as I found it useful in our situation.

We have a Japanese translation in our current game and weren't satisfied in how TMP breaks lines in Japanese. I've created a system that uses MeCab to analyze the text and automatically insert <nobr> tags and zero-width spaces to make TMP break the lines where we want.

I've considered a few options for where to integrate this processing. I wanted to keep the original translation files, so the processing could be fixed/improved and then re-applied. But I also wanted it to work seamlessly, without having to manually run it and create some additional artefacts that then could get out of date.

For this, integrating into the YarnProjectImporter was the best option. This way, whenever a translation file is updated, the processing is run automatically when importing the lines into the project.

Unfortunately, there's no way to extend the current project importer to do some additional processing on the translations. So I've modified the importer to allow just that:

/// <summary>
/// Process Yarn localizations as they are imported.
/// Requires modification to the <see cref="YarnProjectImporter"/>.
/// </summary>
public abstract class YarnLocalizationProcessor
{
    /// <summary>
    /// The language id to process.
    /// </summary>
    public abstract string LanguageId { get; }
    /// <summary>
    /// Process the localization entries.
    /// </summary>
    public abstract IEnumerable<StringTableEntry> Process(IEnumerable<StringTableEntry> entries);
}

In YarnProjectImporter.OnImportAsset, before iterating languagesToSourceAssets:

// Find and create localization processors
List<YarnLocalizationProcessor> processors = null;
var processorTypes = TypeCache.GetTypesDerivedFrom<YarnLocalizationProcessor>();
if (processorTypes.Count > 0) {
    processors = new List<YarnLocalizationProcessor>();
    foreach (var type in processorTypes) {
        processors.Add((YarnLocalizationProcessor)System.Activator.CreateInstance(type));
    }
}

In YarnProjectImporter.OnImportAsset, after parsing the CSV:

// Run localization processors
if (processors != null) {
    foreach (var processor in processors) {
        if (processor.LanguageId != pair.languageID)
            continue;
        stringTable = processor.Process(stringTable);
    }
}

What do you think, is this something that could be useful to merge?

Of course, the interface does not need to work exactly like this, that's just something I created ad-hoc to get it working. One possible improvement could be to use AssetImportContext.DependsOnCustomDependency to allow versioning the processors independently of the project importer.

[McJones]

This is a cool idea, I think my main thought is any reason why this isn't better done as a line provider subclass? Line providers are already intended to be customised based on the needs of your game.

So in your custom line provider subclass when it gets its call to PrepareForLines call you could do something like:

override void PrepareForLines(IEnumerable<string> lineIDs)
{
   if (processor.LanguageID == this.textLanguageCode)
   {
      // do custom processor stuff here
   }
}

Or even at presentation time by overriding GetLocalizedLine and do it on a line by line basis instead.

Thoughts?

[McJones]

Upon reflection we feel this is a presentation concern, not an import concern so a line provider is the best way forward. Thanks for the issue.