search

NicolasTurpin / Touhou-Gensou-Mahjong-4N---ENGLISH-translation

A repository to share the patched files and adress feedbacks
11 stars 5 forks source link

Script error when starting a single game #1

Closed remi-sc closed 2 years ago

remi-sc commented 2 years ago

To reproduce:

Right after starting, before any tiles are drawn I am presented with following error message: Screenshot_2022-02-03_08-50-19

NicolasTurpin commented 2 years ago

Aaaaaaaaah let me guess something : is your game set to the latest official version ? two ways to know :

The game never updates itself with the updater and D.N.A. Softwares released a manual patch on their website that add and fixes a lot of things. My translation is based on this latest patch.

It can be downloaded here : (the green link inside the core text, it should download you "thmj4n.ManualUpdate.exe") https://www.dna-softwares.com/article/2128

How to use it :

Here you go, your game is patched to the last version and Then ONLY you can copy my patch !

Hope it will solve the issue !

remi-sc commented 2 years ago

in the title screen, at the bottom left, you are supposed to have "Ver. 201408302235"

oh, ok, will update and check in a few hours when I have more time, I was under impression that since the updater didn't "update" anything then it was already the latest version

NicolasTurpin commented 2 years ago

It seems like it, I just checked and yes, the magnification are now in strings and not float (for an unknown reason) but the updated version likes it this way were the un-updated seems to want floats only x)

It worked for me and a friend with a brand new install of the game so I guess that's just it ! If this works for you I'll do a proper installation guide in the repo's root including a tutorial for the manual update !

EDIT: And yeah the update brings you a lot of cool new stuff so that's always good to have hehe It seems that the game didn't update depending on a lot of strange stuffs (like your PC's region) but I'm not sure... I'm just glad they provided a manual version ^^

NicolasTurpin commented 2 years ago

Last thing : If you try anything, do it with my 01.1 version : I fixed a bug I caused with the achievements (a dumb omission) I re-tested it and it works like a charm now ^^

remi-sc commented 2 years ago

Updating didn't help.

This was the order of events:

  1. Start with vanilla unaltered fresh installation.

  2. Started it and checked the version: 201405222018 md5 sums of files at this point:

    e829a1567d39d0bccd6e330e0f588982  cri_ware_pcx86_LE.dll
8a9951d0a95bc23e5064256755cf495b  thmj4n.exe
0df722ef64d0502c2e9f0d153c1bcb42  thmj4n.p
61335de46a4df2b26e12ec935fe416ce  updater.exe
6fd4ce1a369e2f15a69b55daae1239c1  updater.p

  3. tried running updater.exe - no files changed

  4. run thmj4n.ManualUpdate.exe. update log:

    差分ファイル [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.ManualUpdate.exe]
差分ファイルの整合性をチェックしています...
差分を適用するフォルダ [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\]
差分ファイルの情報を取得しています...
旧バージョンのファイルを確認しています[MD5チェックサム]...
--> [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.exe] MD5チェックサムを計算中...
--> [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.p] MD5チェックサムを計算中...
旧バージョンのファイル内容が異なります。次の差分データの適用を試みます。
差分ファイル [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.ManualUpdate.exe]
差分ファイルの整合性をチェックしています...
差分を適用するフォルダ [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\]
差分ファイルの情報を取得しています...
旧バージョンのファイルを確認しています[MD5チェックサム]...
--> [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.p] MD5チェックサムを計算中...
旧バージョンのファイル内容が異なります。次の差分データの適用を試みます。
差分ファイル [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.ManualUpdate.exe]
差分ファイルの整合性をチェックしています...
差分を適用するフォルダ [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\]
差分ファイルの情報を取得しています...
旧バージョンのファイルを確認しています[MD5チェックサム]...
--> [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.p] MD5チェックサムを計算中...
[Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.p] 更新元ファイルはアップデート済みです.
差分ファイル [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.ManualUpdate.exe]
差分ファイルの整合性をチェックしています...
差分を適用するフォルダ [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\]
差分ファイルの情報を取得しています...
旧バージョンのファイルを確認しています[MD5チェックサム]...
--> [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.exe] MD5チェックサムを計算中...
--> [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.p] MD5チェックサムを計算中...
--> [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\updater.exe] MD5チェックサムを計算中...
--> [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\updater.p] MD5チェックサムを計算中...
[Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.exe] 更新中です...
[Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.p] 更新中です...
[Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\updater.exe] 更新中です...
[Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\updater.p] 更新中です...
差分ファイル [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.ManualUpdate.exe]
差分ファイルの整合性をチェックしています...
差分を適用するフォルダ [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\]
差分ファイルの情報を取得しています...
旧バージョンのファイルを確認しています[MD5チェックサム]...
--> [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.exe] MD5チェックサムを計算中...
--> [Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.p] MD5チェックサムを計算中...
[Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.exe] 更新中です...
[Z:\home\remi\Games\Touhou_Unreal_Mahjong_4N\thmj4n.p] 更新中です...
----------
201404272232 → 201405212151 の差分をスキップしました。
201405212151 → 201405221949 の差分をスキップしました。
201405221949 → 201405222020 の差分を適用しました。
201405222020 → 201408072135 の差分を適用しました。
201408072135 → 201408302235 の差分を適用しました。
----------
差分ファイルの適用が終了しました.

    md5 sums after updater is finished:

    e829a1567d39d0bccd6e330e0f588982  cri_ware_pcx86_LE.dll
143b11e81ed55f71d361caa712ead9f9  thmj4n.exe
966380c7f6de7f02249fb07c7cb04335  thmj4n.p
08b07f1670842ce79f1a6102fc3ee472  updater.exe
48058e95f6c95d3c1e77579d8a242216  updater.p

  5. Started it and checked the version: 201408302235

  6. Tried your translation files - I am getting the same error on both 01 and 01.1

  7. Revert to original files (from point 5)

  8. run updater.exe. Now some of the files did change!: md5 sums after updater.exe finished:

    677af55efb1ec0bf42ea671b7e06d3b1  cri_ware_pcx86_LE.dll
7dabc0b62591a271e220619ee8725620  thmj4n.exe
966380c7f6de7f02249fb07c7cb04335  thmj4n.p
08b07f1670842ce79f1a6102fc3ee472  updater.exe
48058e95f6c95d3c1e77579d8a242216  updater.p

  9. Try your translation files - I am getting the same error on both 01 and 01.1

There are a few things that may have gone wrong: a) The 201405222018 base version is somehow causing incorrect patching - note that the updater log does not include it:

201404272232 → 201405212151 の差分をスキップしました。
201405212151 → 201405221949 の差分をスキップしました。
201405221949 → 201405222020 の差分を適用しました。
201405222020 → 201408072135 の差分を適用しました。
201408072135 → 201408302235 の差分を適用しました。

This version was mentioned in the comments on https://www.dna-softwares.com/article/2128 But I don't really see anything insightful there. + the updater log seems to check the md5 sums itself so corruption is unlikely b) I am using wine on Linux instead of Windows. Will try to check on windows.

Note: updated vanilla japanese version works correctly - only after patching with your thmj4n.p I see an error.

@NicolasTurpin : Could you please post your vanilla unaltered 201408302235 md5 file checksums that you are using?

Edit: Just to clarify: I am running the whole thing in a git repository so I would have caught any unintentional file changes.

remi-sc commented 2 years ago

b) I am using wine on Linux instead of Windows. Will try to check on windows.

Yup, that is the issue. On native windows machine everything works. And patching on windows and running on Linux does not help.

wine doesn't seem to like your patch.

I already know how to unpack/decrypt thmj4n.p Could you could tell me how to repack the files back?

I would try to fix those wine issues - Since I am not willing to go back to windows...

remi-sc commented 2 years ago

Could you could tell me how to repack the files back?

ok, nevermind, just running LPACK.exe {path_to_dir} is enough. I will try to look at those wine issues this weekend.

NicolasTurpin commented 2 years ago

I'm checking it ! Thanks a lot for the report and all the logs !

I'm not that much informed about Wine but would the problem comes from the cri_ware_pcx86_LE.dll or the d3dx9_35.dll ?

Okay so I'll sum up what I've understood. from your install of the game you get the following :

JP (original)        on         Wine                 : Working
JP (original)        on         Windows              : Working
EN (translated)      on         Wine                 : **Failing**
EN (translated)      on         Windows              : Working

Is that correct ?

EDIT: strangest thing is that you tested all of this with the manual update so I don't see why the combinaition Wise + EN would fail, I'm trying to think of all possibilities but that would point toward md5 issue which I'm not familiar with. And even so, why would it work on Windows ? That's so strange ^^" If that can help, how do I give you my md5 cheksum ?

2nd question : In any case, the game managed to load up to the title menu but only failed when starting a match ?

3rd question : Are we clear this have nothing to do with the computer Unicode language settings ?

NicolasTurpin commented 2 years ago

If that can help, how do I give you my md5 cheksum ?

Nevermind, I found it x) I should always try before ask, sorry haha

On the 201408302235 JP-original version (+manual update), I got this :

thmj4n.p                                966380c7f6de7f02249fb07c7cb04335
thmj4n.exe                              143b11e81ed55f71d361caa712ead9f9
cri_ware_pcx86_LE.dll                   e829a1567d39d0bccd6e330e0f588982
updater.exe                             08b07f1670842ce79f1a6102fc3ee472
updater.p                               48058e95f6c95d3c1e77579d8a242216
thmj4n.cfg                              3f27d4eae9ce05a7a232bf346693ae1d
updater.cfg                             d5f7eaa47dea70e9c75a436849ba0ccc
emotedriver.dll                         bc6b8cb587daf307ac987b8590f18022
xdelta3.exe                             70707830234212e86fb311f49be53459

all the same as yours !

PS: what do you mean by this ?

a) The 201405222018 base version is somehow causing incorrect patching - note that the updater log does not include it:

because of the 2 first skips ?

NicolasTurpin commented 2 years ago

Sorry for the metric fuck tonne of post-edits in my messages ^^"

NicolasTurpin commented 2 years ago

2nd question : In any case, the game managed to load up to the title menu but only failed when starting a match ?

IF ULTIMATELY the issue on Wine+EN is still the same as your original post : I can still convert every "magnification" from string to float and get the job done !

We would still don't understand why this happenend but if it works that's cool x)

remi-sc commented 2 years ago 
JP (original)        on         Wine                 : Working
JP (original)        on         Windows              : Working
EN (translated)      on         Wine                 : **Failing**
EN (translated)      on         Windows              : Working

Is that correct ?

Correct

2nd question : In any case, the game managed to load up to the title menu but only failed when starting a match ?

Yes, all versions launched correctly, only the translated version on wine failed when starting a match

3rd question : Are we clear this have nothing to do with the computer Unicode language settings ?

I highly doubt it - I launched all wine versions with the same Japanese language setting and japanese timezone - only the translated version failed. But I can easily check other locale settings.

PS: what do you mean by this ?

At this point it does not matter. For a moment I thought that since the auto-updater didn't mention my specific version then it could have patched with unexpected corruption as a result. That was before I confirmed that only wine was affected. + All your checksum match mine. So you can forget that I said that.

IF ULTIMATELY the issue on Wine+EN is still the same as your original post : I can still convert every "magnification" from string to float and get the job done !

If it's not much work then please, go ahead, I planned to try doing that since debugging wine while possible isn't something I am very familiar with.

remi-sc commented 2 years ago

Actually it might have something to do with locale after all. A similar issue was described here: https://gitmotion.com/pathofbuilding/179640315/builds-for-other-platforms With a possible explanation being different numeric locales using dots or commas for numbers which could make lua misbehave when converting string<>float

NicolasTurpin commented 2 years ago

Note : I tried launching the game on my Windows French Unicode AND Japanese Unicode, it worked in both cases.

Tomorrow I'l give you the file file the "magnification" in floats, and we'll try this out ;D

remi-sc commented 2 years ago

Thank you, Not entirely sure what yet which of your changes could have caused this. I guess I will simply try different locale settings and if that doesn't work I can also go through your changes and try applying them one by one until I find the culprit.

NicolasTurpin commented 2 years ago

CHARACTERS.CSV

Feck, I'll do it right now haha, Here it is ! I converted all string values to float values

You should paste it in "S/STATS/" if you know how to repack everything (else I could repack myself and do a v01.2)

EDIT: just for the test, take 4 Reimu Players in case i forgot one ^^"

PS: I'm 100% sure this was the thing that caused your original bug ! but it can also be something else added to the mayhem

NicolasTurpin commented 2 years ago

Actually it might have something to do with locale after all. A similar issue was described here: https://gitmotion.com/pathofbuilding/179640315/builds-for-other-platforms With a possible explanation being different numeric locales using dots or commas for numbers which could make lua misbehave when converting string<>float

And also also, Yeah the local settings are changing a few things. Not enough to break the game but it can mess up your "Tweets" the game allows you to send and the save folder.

remi-sc commented 2 years ago

Feck, I'll do it right now haha, Here it is ! I converted all string values to float values

Works! I see no more errors. Going to sleep now, will do more tests and try to find any translation errors tomorrow after work.

NicolasTurpin commented 2 years ago

I guess you're more British than American as for the time you're going to sleep xD (don't answer this I'm just having fun)

It's nice if it works !! I guess the game was calling for OS dependent operation for some strange reason and the conversion from string to float was messed up by Wine (I'm just guessing but that's still strange)

Anyway it works even more so I'm hoping for the best !! Have a good night !

remi-sc commented 2 years ago

After going through the lua files while looking for translation errors (#2 ) I am now much more inclined to believe that the wine problems are with locale. I can't even find an editor that would display this mess correctly. And manually setting encoding to euc-jp/iso-2022-jp/shift-jis/cp932 doesn't seem to match those files. Not sure what is the correct encoding.

NicolasTurpin commented 2 years ago

I actually set the encoding to Japanese Shift-JIS just like you !

What's weird is that the unmodified game works with Wine

Would that mean... that the messing modification don't comes with what I've written and more with the way TheKeithD scripts re-encode the files ?

If that's the case, you can try decode the original files, repack them and check if nothing changes !

remi-sc commented 2 years ago

I didn't get to spend much time on the issue this weekend but I did find a few basic things:

NicolasTurpin commented 2 years ago

Don't worry I was quite busy too this week-end x) I managed to finish all the Training mode translation that I left aside for too long, now there's just the status effects to do !

One with "lunpack -b" and second with "blowpack -d"

I guess I was just lucky to have chosen to use "blowpack-d" then x)

just changing a comma to a dot there while preserving string types is also enough

That's super nice to know ! It will be less of a pain if I have to change this csv again ^^

Changes that removed all japanese characters from a given file could have changed encoding auto-detection.

That's actually super clever, I didn't thought about this and it actually might have happened on very small files, supposing there is not any dev comment. The thing is S/SCENE/SIDEBAR.LUA don't contains any Japanese character since the beginning, wouldn't it mess with the Vanilla version too if that was the issue ?

NicolasTurpin commented 2 years ago

But anyway if it works fine now on Wine that's great !! We are covering several OS haha

remi-sc commented 2 years ago

I managed to mostly figure out what is going on. Basically there are a few main findings:

To elaborate:

  1. Depending on the "numeric" locale detected by lua engine a decimal separator can either be a dot or a comma: https://www.lua.org/pil/22.2.html Non-matching config should result in errors (but there seem to be exceptions to this - see point 4.)
  2. One of my ideas for what is going on is not what is happening:

    Changes that removed all Japanese characters from a given file could have changed encoding auto-detection.

Just changing the vanilla game files from floats to strings with a comma in S/STATS/CHARACTERS.CSV is enough to cause an issue.

  1. To reproduce the issue with "comma strings" in CHARACTERS.CSV on windows the "format" has to be changed to one which uses a dot. For example Japanese. (Yes - version with strings using a comma seemingly won't normally work on Japanese system by default) Format can be changed in control panel > clock language and region > region and language > "format" tab > change format to Japanese or other that use a dot. (https://en.wikipedia.org/wiki/Decimal_separator#Countries_using_decimal_point)

I initially checked on a machine with Polish format - which uses a comma. I also presume you used French - which is also on the "comma list" This would be why it worked

  1. I tested a few CHARACTERS.CSV versions:

    version:                   | wine (LANG=ja_JP.UTF8) | windows PL format    | windows JP format
---------------------------+------------------------+----------------------+-------------------
using floats               | works                  | works                | works
using strings with dots    | works                  | works (not sure why) | works
using strings with comma   | doesn't work           | works                | doesn't work

  2. problems with "comma strings" on wine can be worked around by in least one of 2 ways:

    • change a registry value to a comma: HKEY_CURRENT_USER\Control Panel\International\sDecimal (run 'wine regedit')
    • Launch wine with a locale that uses a comma decimal separator. For example LANG="pl_PL.UTF8" (but this may result in text rendering errors in some places)

  3. As an alternative to changing the CHARACTERS.CSV back to floats, it may be possible to use os.setlocale(..., "numeric") in BOOT.LUA : OnLoad() - to pre-set the expected numeric locale. The game already does this for "time" locale. But I did not test this.

At this point I think reverting to floats in CHARACTERS.CSV would be the easiest and most universal solution that wouldn't involve any system specific tweaks.

@NicolasTurpin A question however arises: originally there were floats there, why did you change them to strings in the first place? I don't really see a place where "magnification" would be used as something other than a number. If there is a reason to leave them as strings then let me know. I may try to investigate the "os.setlocale" avenue.

NicolasTurpin commented 2 years ago

Wooooh okay ! That's very strange, I never expected countries to have their own comma VS dot thing x)

I actually never tried to change the os.setlocale in th BOOT.LUA but it can be a very nice thing to do ! The game actually fails to interpret some japanese characters in specific situations (naming the main window, sending a tweet, etc... maybe even custom PR titles my appear broken to JP players) So when I wanted to play online I launched some EmuLocal app, but that's quite a pain if everybody needs to set their local to JP. I'll test the BOOT.LUA !!!

originally there were floats there, why did you change them to strings in the first place?

Oh... FECK !! you're right !! It seems that the software I first use to modify the CSV did that convertion. I feel so dumb, I never checked the original file :s

I reverted it to float some time ago but that's a real shame all of this comes down to my mistake

remi-sc commented 2 years ago

that's a real shame all of this comes down to my mistake

Don't worry about it. The important thing is that with the floats back in their place the issue is effectively solved for now. And we have a better understanding of potential issues.

NicolasTurpin commented 2 years ago

I tested to set the os.setlocale to Japanese

os.setlocale("Japanese_Japan.932", "all"); os.setlocale("English_America.932", "time"); I printed the result in the window caption 'cause I'm a shlagg haha setWindowCaption(os.setlocale(nil, "all")) -> return all the locale(s) image

It worked, in a way, but is still unable to render the "out of the game text" correctly : The create-room tweet still give me garbage instead of the few JP characters I left

“Œ•ûŒ¶‘z–ƒ4N Touhou Gensou Mahjong 4N - Room Name: Open Table (Allow Magic,East Only) ƒNƒCƒbƒNƒ}ƒbƒ` ‚ÅŽQ‰ÁŽó•t’†I Is accepting participation ! (01:53)

same goes for the starting window caption

Note: Japanese_Japan.932 is the locale I have when setting the game to Japanese with EmuLocal (which works and render all characters properly)

NicolasTurpin commented 2 years ago

Well I guess there's nothing we can do about this except recommand Twitter users to use a local emulator x)

Anyway I left the intern locals to Japanese with English time (as in the previous post) and re-uploaded the last version (with a pretty title background brought back from THMJ4 hehe)

If you have time to test the Story mode, tell me, there is some surprises inside ;) you can tweak you save.dat with blowpack.exe -d if you want to skip a lot of them by setting "storylevel" to 29 (which will allow you to test the last bonus levels)

We didn't had time to do any last testing approval with my friends so the official release is pushed to next week-end I guess. Have a nice 14th Feb' and see you later ;D

remi-sc commented 2 years ago

I think you are looking for "os.setlocale("Japanese_Japan.932", "all");" or "os.setlocale("ja", "all");"

Edit: nevermind. Somehow github didn't load all your edits and I didn't see them

There is no shift-jis locale available there - since AFAIK lua doesnt really handle character encodings. (shift-jis is only an encoding, not a whole locale)

If given shift-jis string isn't working then maybe it is simply pushed to a library that doesn't use shift-jis? Could you point me to one of such strings? I can take a look.

Also:

It seems that the software I first use to modify the CSV did that convertion.

There are some unexpected commas added at the end of some lines in S/STATS/CONFIG.CSV Was that intentional or also a mistake?

NicolasTurpin commented 2 years ago

There are some unexpected commas added at the end of some lines in S/STATS/CONFIG.CSV

Arh nope, unintentional ! I don't think that's an issue tho, but I assume my software added them due to the empty collumns.

Could you point me to one of such strings? I can take a look.

Absolutely ! Here are some strings that are rendering well on my JP local setup but are all messed up on my FR local :

\BOOT.LUA line 197 setWindowCaption("東方幻想麻雀 4N+");

\S\SCENE\MULTI\LOBBY.LUA line 641 local tweet = string.format("東方幻想麻雀4N Touhou Gensou Mahjong 4N - Room Name: %s (%s,%s) %s で参加受付中！ Is accepting participation ! %s(%s) #thmj4n", roomname, usemagic, length, password, comment, os.date("%H:%M"))

\S\ALGORITHM\SHOWAGARIRESULT.LUA line 361 local gamemode = string.format("%s :%s Completed (on %s - %sP with : %s)", SERIAL:get("name"), yakuname, TAKU.jname, PLAYER_MAX_NUM, cname);

the previous line is refering to yakuman names scattered all around here \S\ALGORITHM\HAI_CODE.LUA line 857+ yakus:yproc(seat,eloc,_yaku_kokushi,"国士無双",13,1,"kokushi");

And the savefiles path which seems to be contained in the .exe more than the lua files (? I can't manage to find the full save path concatenation line, but that's not very important haha)

That's all I've found but I don't know if JP people are seeing our titles like garbage or not (as it is saved in a string, not an achievement index) I'll test this with friends, I think they all have achievements by this time haha

NicolasTurpin commented 2 years ago

As a note, I wanted to leave some japanese in the create_room tweet so that non-english-speakers can still see a welcoming message and join EN rooms... but if that's impossible, I can just remove it. Same thing for the Yakumans, I can just put them in Romaji and call it a day. As for the WindowCaption, I already modified the one appearing during all the game, I can also modify this one (I just left it this way to now when you launched the game on a JP local or not)

EDIT: Well, feck it, I only put the Romaji name of Yakumans in the tweet x) should have done that earlier. But It would still be nice to have a bilingual create_room tweet

NicolasTurpin commented 2 years ago

Uuuuuh We'll not tell anyone but did you do something to break the server while testing ? ^^"

EDIT: Okay no, weird, all is working despite an error message appearing on the Telop, even in vanilla. But the room are creating normally, it must have been a coincidence ^^"

remi-sc commented 2 years ago

We'll not tell anyone but did you do something to break the server while testing ? ^^"

No, Coincidence. (I am not testing anything since yesterday)

remi-sc commented 2 years ago

Absolutely ! Here are some strings that are rendering well on my JP local setup but are all messed up on my FR local :

\BOOT.LUA line 197 setWindowCaption("東方幻想麻雀 4N+");

I don't think there is much that can be done here besides writing logic around various detected locales. The window title encoding changes depending on the locale. And for some reason it isn't what I would expect.

For example if we wrote setWindowCaption("東"); and saved in existing shift_jis locale that lua file will contain a hex bytes of "東" - "0x938c"

When those bytes are pushed somewhere to winapi to set the title they are interpreted according to given locale. So depending on a locale set we would get ôî (IBM850), “Ś (windows-1250) or some other mess. (To add to this mess for example using pl_PL.UTF8 LANG actually results in windows-1250 title encoding instead of UTF8)

Since we can't really control the locale of users and we don't have access to whatever is actually calling winapi the only thing I see that could be done is to detect the locale, convert the original shift_jis to appropriate encoding and push that converted one out.

This isn't really a solution however because even if there was a converter for those many encodings there would still be an issue that some characters simply do not exist in a given encoding. And you would need to stick to non-japanes characters.

For those tweet issues however: I think you could just try to force UTF8 - I assume twitter expects something sane like that.

So instead of this...: local tweet = string.format("東方幻想麻雀4N Touhou Gensou Mahjong 4N - Room Name: %s (%s,%s) %s で参加受付中！ Is accepting participation ! %s(%s) #thmj4n", roomname, usemagic, length, password, comment, os.date("%H:%M")) ... check if this hard-coded UTF8 tweet message would appear correctly. local tweet = string.format("\xe6\x9d\xb1\xe6\x96\xb9\xe5\xb9\xbb\xe6\x83\xb3\xe9\xba\xbb\xe9\x9b\x80 4N Touhou Gensou Mahjong 4N - Room Name: %s (%s,%s) %s \xe3\x81\xa7\xe5\x8f\x82\xe5\x8a\xa0\xe5\x8f\x97\xe4\xbb\x98\xe4\xb8\xad\xef\xbc\x81 Is accepting participation ! %s(%s) #thmj4n", roomname, usemagic, length, password, comment, os.date("%H:%M"))

if that works then maybe a shift_jis to utf8 helper function would be handy.

NicolasTurpin commented 2 years ago

That's nice to have someone who is fluent in these things ^^

At least we are now certain there's nothing we can do, so I just removed all japanese characters from the tweets. I mean... they are just tweets, if you really want to have Japanese people joining, you can either write your own tweets or bet on JP players that can understand a simple message like this. At least they'll receive the notification. EDIT: OR you can put a JP appealing message in the description of your room, appearing in the tweet !

Oh wait !!!

I should read your message in their entierty before writing haha I'll try your like proposition !! If it works that's insane !

NicolasTurpin commented 2 years ago

My Franco-Japanese friend is actually helping me write a proper japanese message for the release so while waiting I'm trying to refacto the "Replay Mode" unfinished by the devs haha

Did you had time to test the story mode ? ^^

NicolasTurpin commented 2 years ago

I'm recompiling with your UTF8 idea, while waiting, can I ask you a personnal question ?

Why ? or maybe What made you help me this much ? :3

NicolasTurpin commented 2 years ago

Ooooh ! It worked to some extend !

trying this part : で参加受付中！

before, with japanese encoded characters : ƒNƒCƒbƒNƒ}ƒbƒ ‚ÅŽQ‰ÁŽó•t’†�I <- there's a " ` " in the middle

now with the bruteforce UTF8 : で参加受付中！ (changing the inside os.locae to Japanese or not gives the same result)

maybe I missed something, what did you mean by :

if that works then maybe a shift_jis to utf8 helper function would be handy.

NicolasTurpin commented 2 years ago

Last message for today :

You can open a new issue for anything related to Story Mode / Achievements / Twilight Bar (if anything haha)

remi-sc commented 2 years ago

ã�§å�‚åŠ å�—付中ï¼�

welp, this looks like it was decoded assuming french/western encoding ISO-8859-1 You can try it for example here: https://string-functions.com/encodedecode.aspx (paste で参加受付中！ into the box, select encode with utf-8 and decode with ISO-8859-1)

Which means that somehow that twitter handling code is also expecting french encoding. And since ISO-8859-1 does not support japanese characters there isn't much you can do at this moment (outside of removing those characters).

Did you had time to test the story mode ? ^^

Nope, I didn't even try launching it yet.

Why ? or maybe What made you help me this much ? :3

I'm not really doing that much honestly. At first I just wanted to check how much work would it be to translate just table rules since those were ambiguous to me. I saw a repository with tools ready. Then key extraction was just a good sport. I saw you asked for it so I gave it to you. I'm just helping with a game I like.

maybe I missed something, what did you mean by :

if that works then maybe a shift_jis to utf8 helper function would be handy.

It was just a suggestion that if that hardcoded utf-8 string worked then instead of translating manually and inserting those \xAB\xCD\xEF raw bytes it would be easier to just have a helper function. Now that I know it's not UTF-8 but a locale dependent encoding (ISO-8859-1 for french) then this is irrelevant.

(changing the inside os.locae to Japanese or not gives the same result)

This is expected. As I mentioned somewhere above: lua doesn't touch character encodings. all it does is read the string literal bytes and push them out. So without any conversions the output bytes are fixed at the moment you save the .LUA file. It's just how those bytes are interpreted outside of lua world that matters.

At this point I think a potential solution would be to somehow set the locale of the exe to japanese using winapi: https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/setlocale-wsetlocale

If I find time I will try to dig deeper...

remi-sc commented 2 years ago

And I think this issue thread already veered way off-course. Will open a separate one once I have more to add about the encoding/story/something else