search

Interkarma / daggerfall-unity

Open source recreation of Daggerfall in the Unity engine
http://www.dfworkshop.net
MIT License
2.72k stars 329 forks source link

Translation support #1393

Closed petchema closed 3 years ago

petchema commented 5 years ago

(Maybe this should be labelled "task" instead of "enhancement", will see)

The roadmap contains two entrties:

The first one especially looks like repetitive work, so if I'm right this is totally something that anybody (including me) could help with, with some coordination so:

petchema commented 5 years ago

I figured that translation teams must have translation tables for those strings, so keeping them in a single file may actually make their life easier

Nystul-the-Magician commented 5 years ago

first step is to get rid of remaining hardcoded texts that are not already in HardStrings and move them there (e.g. automap tooltips - I will do this as next step for the classes I know) so a thing to do would be to search for hardcoded strings outside HardStrings but used in-game as text elements

update: wait after taking a look, it is better to move them directly to a .txt text file ;)

ajrb commented 5 years ago

I'm in the process of doing the hardstring removal for guilds & services. I figured this would happen piecemeal and happy to move any I created/used.

Interkarma commented 5 years ago

On my end, this roadmap item has a several clearly defined sub-tasks. I have already spent a fair amount of time planning this out. I'll enumerate what I have in mind below.

  1. Extended font support. Move away from classic 256 glyph limitation to an expanded atlas format with any number of characters, improved packing, and peer XML file for mapping characters to glyphs, kerning hints, etc. Unicode support is a must, as is easy generation of new font. At this time the plan is to create a new font type in DagUI supporting both SDF and standard pixel fonts while removing the current limitations. Evaluating Littera for font generation.
  2. New unified text storage standard. Move away from .txt files and my table format to something like JSON for text storage. Using a standard like JSON allows any kind of front-end to edit text data. This can be an editor UI in Unity or any kind of forms application. This storage standard will need to support text arrays, etc. as seen in HardStrings. Also provide export mechanism to create a dump of all in-game text in this new format. A unified English text database will be created that will contain standard in-game text and provide an archetype for future translations. The English text database will also act as a fallback for any text not in other translations, allowing this work to continue piecemeal at whatever pace the community involved prefers.
  3. Update text API to read from new unified standard. I've made a few experiments in text over the years and would like to bring all of these back into line. Aiming to make this as smooth and clear as possible.
  4. Experiment with self formatting text (automatic line breaks, etc.). This one is more of a longer-term target, but is valuable for languages that don't fit nicely into the strictly defined formatting of classic. This would allow translators to simply write content without needing to muck about trying to format text. This would also allow for improved text proportions and consistency compared to now.

Some of those are obviously easier than others. I had planned to start work on items 1 & 2 as a priority while developing 3. Then start migrating all text in game to new system. Item 4 is more of a UI task and I see this as a refinement that should emerge naturally by the time everything else is done.

I figured that translation teams must have translation tables for those strings, so keeping them in a single file may actually make their life easier

Agree completely. My vision of next text upgrade is to move to a single file storage that can be injected by mod system. Then translators can pack together fonts, audio, text database, and replacement UI images into a single .dfmod.

Interkarma commented 5 years ago

I should add that quests will remain in a single .txt script for ease of quest development, along with their original text. But the unified text database will be able to override quest text as needed. This way translators don't need to worry about the QBN part of script and just override QRC parts externally.

Nystul-the-Magician commented 5 years ago

Is there a way to use texts with a comma char inside it when using the text database files? If I try it it breaks text resolving since the comma is also the separator character used in this .txt files - can one escape it somehow?

Interkarma commented 5 years ago

Is there a way to use texts with a comma char inside it when using the text database files?

Yes, the text tables can do this with the following schema:

schema: *key,$text

The $text means this is a string literal and supports commas. You then just have to wrap the text data in inverted commas like so:

canOnlyCastOncePerDay, "Can only be cast, like, once per day, man..."

See the GeneralText.txt file for an example. This support came along a lot later which is why not everything uses it. I guess not everything needs it either.

The unified JSON text database I mention above will be able to store all text in the game. It will be a replacement for the 12+ places text is stored now between classic and DFU. I want to bring everything down to a single standard file format in this phase.

Nystul-the-Magician commented 5 years ago

thanks for the help ;)

question: I would like to put the automap related stuff to DaggerfallUI.txt - but I would need to change the schema there, is it ok to do so? I would add the quotes for all entries there...

Nystul-the-Magician commented 5 years ago

pull request (https://github.com/Interkarma/daggerfall-unity/pull/1394) issued which migrates all hardcoded texts from automap and exterior automap code to DaggerfallUI.txt (changed schema to string literal schema in this file and added quotes for all entries)

ajrb commented 5 years ago

Okay so you're planning a hell of a lot more work than I thought you would... thought ya wanted to move on. lol Anyway, guess I'll put the hardstring migration I've been doing piecemeal on hold. May as well wait for the new formats to be in place first instead of having to move stuff twice.

Interkarma commented 5 years ago

I'll do something to migrate the data tables automatically to new text database, it won't be completely wasted time. But still easiest just to wait a bit for now. :)

deepfighter commented 5 years ago

Funnily, I wrote two weeks ago to Interkarma and asked him about his plan of translation support, too. For consistence reasons I will put my thoughts here as well. I can't help much on the technical part, but I want to signalize that I would love to test translation support with the German version if implemented, as we finished for Classic Daggerfall with everything except for some quests, main screen picture and some parts of TEXT.RSC. So please do give me a signal if you need my assistance - I would be glad to help where I can. :)

Regarding my thoughts - what I would like is a common approach for future translations, which everyone has to follow. I found four options I would like to share with you:

  1. Translators fork DFUnity completely and change everything which can be translated (pictures, text strings,...) and release it as stand-alone version;
  2. Translators fork DFUnity only in these parts which are needed to be translated (pictures, text strings,...) and release it as kind of a "patch" which can be applied on top of an existing DFUnity translation and is only indirectly connected to the DFUnity Repo;
  3. DFUnity implements a language option, which means DFUnity has the same files in different languages, filtered in languages folders. E.g. StreamingAssets/Text/DE, StreamingAssets/Text/ES,...and you can pick before starting DFUnity which language you want to play the game;
  4. Combination of 2 and 3, make one language repo, push that into corresponding folders of the dev-repo.

As I not totally understand the technical concept I am not sure which is the most feasible option. Of course, to give other languages some visibility in the long-term, the third/fourth option would be an jackpot, and as far as Interkarma told me he wants this to be "as easy as dropping in a mod file and picking language in settings", which would be like my option 3.

We should definitely avoid option 1 as the risk is high that these translations don't merge current changes from the dev-repo and offer an old DFUnity version. So we need a way to establish that technically only one version of DFUnity exists, but maybe also give the possibility to have control over the translation - specifically maybe have additionally an own language repo e.g. for issue tracking. To share these repository publicly would also have the advantage that "abandoned" translations (which will in any way happen for some) could be picked up by others without hassle.

petchema commented 4 years ago

I just came across that specification pushed by Mozilla: https://projectfluent.org/ Sadly it doesn't seem there's any C# implementation, but it could at least be food for thoughts

rossipaolo commented 4 years ago

I just came across that specification pushed by Mozilla: https://projectfluent.org/ Sadly it doesn't seem there's any C# implementation, but it could at least be food for thoughts

Maybe the Python implementation can be used with IronPython?

Interkarma commented 4 years ago

I'm sorry I haven't been able to progress this further in March as promised. My work responsibilities around the pandemic have been higher than expected. Things are returning to a normal rhythm now and I'm reasonably confident I'll be able to start this up again by late April or early May.

petchema commented 4 years ago

Barely anything went according to plans ;)

drLemis commented 4 years ago

There is a format for that, CSV, and you can use I2 Localization for Unity to parse it and get it handle all text (and not only text) changing on interface. Also it's have support for any TMP font, separated for each language.

Interkarma commented 4 years ago

The Unity localisation tools available after recent engine upgrade are quite excellent. I now have most of the core functionality in place. As an added bonus, I don't have to spend hundreds of hours writing bespoke editor scripts

I'll post more about this on Translation forum in a few weeks. I plan to have localisation feature in a preview state for 0.10.26/.27 for translators to start coming to grips with.

petchema commented 3 years ago

Closing this, as I assume it's obsolete now