Characters not supported on translation

[BanCrash]

Hello, I'm making a translation for the new 3.0.0 version and I find that there is 3 characters that aren't supported. I don't know if could be supported, so just in case I let here to know.

Characters are "ú", "Á" and "ü". Third character ("ü") is rarely used in Spanish, so there shouldn't be much trouble if there isn't support for that.

However "Ú", "á" and "Ü" are supported.

By the way, great job with the new version, it looks great!

[Kapiainen]

Characters are "ú", "Á" and "ü". Third character ("ü") is rarely used in Spanish, so there shouldn't be much trouble if there isn't support for that.

However "Ú", "á" and "Ü" are supported.

All of those characters are in the lookup table that is being used, but I suspect the ones causing issues are represented in another way than what Lauhdutin currently supports. I had a quick look around and it seems like plenty of characters with accents and other additional parts can be represented in many ways in Unicode. For example this Stack Overflow thread shows an example of how Ü can be defined in two different ways. Maybe that is the issue?

[BanCrash]

It were my fault. When I saved the txt file, I didn't change Windows default encoding (ANSI). When I set it to UTF-8 the issue were solved. Sorry for the misunderstanding.

[Kapiainen]

Okay, thanks for letting me know.

[BanCrash]

@Kapiainen Seems there is another issue. When I reload the skin, the txt turns again into ANSI encoding. I suppose the skin writes on the translation file on every startup?

[Kapiainen]

It shouldn't do that. Is the first line in your translation file version 1 like in the default English translation file?

[BanCrash]

I didn't change, so should be the same. Should be changed?

https://github.com/BanCrash/Lauhdutin/blob/patch-9/translations/Spanish.txt#L1

By the way, the first line is not working. I put "button_label_browse Examinar", but in settings shows Browse:

I change that line on English.txt to "Browser" instead "Browse" but still no change:

[Kapiainen]

The version should be the same.

I found the cause of that and it will be fixed in the next release. Thanks for letting me know.

[BanCrash]

After using the skin, I see that translation file are changed. Maybe it's related to this issue?

Before use it looks like this:

I found the cause of that and it will be fixed in the next release. Thanks for letting me know.

Perfect, thanks you!

[Kapiainen]

The localization system writes to the active translation file under two circumstances:

1) The translation file's version is less than the localization system's version number. The system has a migration system in place for updating old translation files into whatever new format might be used in the future.

2) The translation file is missing a key-value pair that the localization system is asked for by some part of the skin. The system just writes the key to the end of the file along with TRANSLATION_MISSING (or the default value, if the language is set to English).

[BanCrash]

I found the second option on both translations (english default and spanish).

But I see that line on standar file too:

https://github.com/Kapiainen/Lauhdutin/blob/development-v3/translations/English.txt#L84

Are you also sure that the text editor that you are using always saves the file as UTF-8?

I think yes, because after saving it as UTF-8 I can see settings properly before the next startup of the skin:

Startup:

After saving translation on UTF-8 and reloading settings.ini:

[Kapiainen]

But I see that line on standar file too:

https://github.com/Kapiainen/Lauhdutin/blob/development-v3/translations/English.txt#L84

That was a mistake on my part in the MoonScript source. Fixed in the next release.

[BanCrash]

Everytime I reload the skin, the translation file is modificated. Before starting:

After starting:

However that doesn't happen on english.txt.

So I create another txt, with english default but nammed Test.txt. First, untouched: Before: After that, I unload the skin, wait 3 minutes more or less and load again:

Second test, touched but with default letters (I write "Saveeed" instead "Save"): Before: After:

So seems that when it is a change from the default, the txt is modified. Could be that modification when the problem arises?

[BanCrash]

Yes, I think that is the problem. I had the translation file opened in notepad++, with UTF8 encoding selected, and saved. After loading the skin, notepad++ advise me that the file is modified. I reload the file and encoding get back to ANSI.

[Kapiainen]

I'm not having any luck in reproducing the issue. I tried with a renamed copy of the default English translation and your Spanish translation. The skin wouldn't write to the file every time that the skin was loaded and unloaded. I also tried deleting a line to force the skin to write the missing key-value pair, which it did as expected, but the encoding remained as UTF-8 (according to Sublime Text 3).

[BanCrash]

That's strange. I tried with last beta and with last commit and the issue it's the same in both cases. First time I see the settings correct, but after reloading the skin the problem arises again.

I will make a test with other computer to see if the problem arises there too.

[BanCrash]

In the other computer I have the exact problem. First I see all good but when I reload the skin settings appears bad again.

Did you check if with Spanish.txt, in the second reload settings appears correct? For example, the second line of the first tab of settings should be "Número de filas", and not "Nmero de filas".

[BanCrash]

I set to only-read spanish translation to try to avoid encoding conversion and this is what I get: (it's the last beta, I reuse the beta1 folder for every beta)

But if I make the same with English.txt, nothing happens. All works correctly.

The changes made are:

• The /n converts into a real blank space, as I said before:

• Lines change its place. Before, last line was for initializing, now is for stopping bangs:

Right now I don't see more changes. It is very strange...

[Kapiainen]

I updated the localization system, which should now properly escape newlines when writing to the translation file via the class' save method. I also added log messages to the class, which should help reveal what is going on if you enable logging in the skin's settings. There are separate messages for when the entire translation file is written (the class' save method) and when a missing translation is appended to the translation file (the class' get method).

[BanCrash]

This is what I get I think is relevant to the matter with the last commit:

Full log:

[Kapiainen]

Odd. I only get the Loading translation file regardless of which translation file I use (default, renamed default, or your Spanish translation).

[BanCrash]

https://github.com/BanCrash/Lauhdutin/releases/tag/3.0.0Test

I upload a zip file with all the files I used. Please check if with those files doesn't happen to you this issue either.

EDIT: The first time I start the skin I don't get that message. But the file are converted to ANSI equally. After that, I unload the skin, I convert again to UTF-8 the file, save, and reload the skin, and then appears that log.

[BanCrash]

I solve the issue doing this:

(I converted lines 43-45 of localization.lua to comment)

I also try to remove a line on the translation and the skin writes that line with MISSING_TRANSLATION, but in UTF-8, not in ANSI.

[BanCrash]

After that modification the file is still on UTF-8 on every startup. And in settings, I can see all characters properly. However, game.ini still not work properly.

TXT:

Game.ini:

(ú, Á and ü are not showing).

Same test on settings:

TXT:

Settings.ini:

[BanCrash]

After more testing on this matter (changing 3 random translation to adding them "Áú"), I see that in Game.ini, Filter.ini and Sort.ini are not working those characters either. However, in settings.ini and search.ini are working.

[Kapiainen]

Sorry it has taken so long. I've been banging my head against the wall with this for a while now, but I was able to reproduce the issue and finally fix it, at least on my computer, with this commit. Could you test it as well? You'll need to grab the latest version of the development-v3 branch since the commit is not included in the latest public beta.

[BanCrash]

Awesome! Right now all two issues are solved! The file isn't being writing on every startup now, and I can see properly all the strings!

Unfortunately, seems to appear a new little issue that aren't on last beta (8). But I'd hope it would be easy to fix! In some game names appears some strange characters:

In this case, I see that the proper name is "Batman™: Arkham Knight", so issue is the "™" character.

In this case, proper name is "Age of Empires® III: Complete Collection". So issue would be "®" character.

Thanks you so much for fix this!

[Kapiainen]

I just pushed a new commit that should fix the issue with e.g. ™ and ® characters and make it behave like it did before.

[BanCrash]

Perfect! So this seems to be finally solved! Thanks again for your great work!!

[Kapiainen]

Thanks for your help and patience in troubleshooting and testing.