lmlsb edit improvements

pmrowla commented 5 years ago

lmlsb edit CLI tool only supports a limited subset of TCom commands. Use this thread for discussion regarding support for new commands.

LioMajor commented 5 years ago

Could you please add {stopmovie} to the tcom commands that it doesn't remove it (interprets it wrongly as text)?

pmrowla commented 5 years ago

According to what you put in #10

{STOPMOVIE}

{CHANGECG "動画" "グラフィック\動画\みらい2\余韻.gal" "REPEAT" "1" "1000" "0" "0" "" "1"}

...

this is a novelscript (lns) event and not an lsb TCom command, and shouldn't have anything to do with lmlsb edit.

I'm confused as to what you are actually editing and what happens when you run the patched game?

Are you inserting {STOPMOVIE} into the .lns script, and then the actual text "{STOPMOVIE}" is displayed in the text window in-game?

LioMajor commented 5 years ago

Here is an example of the acutal file:

lmlsb extract 0000000D.lsb
...0000000D-000000A5.lns
...0000000D-000000C5.lns
...0000000D-0000000F.lns
...0000000D-0000009D.lns
...0000000D-00000011.lns

Part of the content for 0000000D-00000011.lns:

;pylm

; Font styles:

;    0: TDecorate(count=254, unk2=4294967295, unk3=4294967295, unk4=0, unk5=0, unk6=0, unk7=4294967295, unk8=, unk9=, unk10=4294967295, unk11=4294967295)

;---------------------------------------

; BEGIN DECOMPILED SCRIPT

;---------------------------------------

<SCENARIO VER="102">

{STOPSND "VOICE" "100" "PASS"}

{STOPSND "SE" "100" "PASS"}

{STOPSND "VOICE2" "100" "PASS"}

{STOPSND "SE2" "100" "PASS"}

{STOPSND "BGM" "1000" "WAIT"}

{DELETECG "動画" "1" "500" "0" "0" "" "1"}

{PLAYMOVIE "スク水フェラM.wmv" "256" "C" "C" "WAIT" "TRUE" "0" "255"}

{STOPMOVIE}

{CHANGECG "動画" "グラフィック\動画\なつみ2\なつみ2_1\余韻1.gal" "REPEAT" "1" "1000" "0" "0" "" "1"}

{PLAYSND "BGM\【陵辱シーンをイメージ】女の子に催眠術をかけて×△○する感じタイプC.wav" "BGM" "REPEAT" "400" "0"}

{PLAYSND "SE\余韻2.wav" "SE" "REPEAT" "800" "0"}

{PLAYSND "VOICE\C16.wav" "VOICE" "REPEAT" "1000" "0"}

<TXSPN>なつみ「ふーっ・・・ふーっ・・・ふーっ・・・ふーっ・・・ふーっ・・・<BR>

　　　　ふーっ・・・ふーっ・・・ふーっ・・・ふーっ・・・」

{PAUSE ""}

<EVENT VALUE="TEXTCLEAR">

<PG>

{STOPMOVIE} is the command to stop playing the wmv.

As lmlsb doesn't recognize it as a command, it thinks of it as text and adds itself before the command and removes where it belongs.

Putting it back inside with lmlsb insert 0000000D.lsb 0000000D-00000011.lns 14 changes the lines what causes the game to break.

 <TXSPN>{STOPMOVIE}{CHANGECG

0000000D.txt

LioMajor commented 5 years ago

Maybe to have an alternative to fixing {STOPMOVIE}, adding lmlsb edit for ingame text will help not to break the script.

Textins: (TXSPD / TXSPF / TXSPN / TXSPS)

Example how it could work:

lmlsb dump 00000001.lsb

 180: TextIns <livemaker.lsb.novel.TpWord object at 0x0000029ABEBA8588> "メッセージボックス" 1 0 0
;pylm

; Font styles:

;    0: TDecorate(count=827, unk2=4294967295, unk3=4294967295, unk4=0, unk5=0, unk6=0, unk7=4294967295, unk8=, unk9=, unk10=4294967295, unk11=4294967295)

; Display conditions:

;    0: TWdCondition(count=892, target=)

; Links:

;    0: TWdLink(count=827, event=, unk3=)

;---------------------------------------

; BEGIN DECOMPILED SCRIPT

;---------------------------------------

<SCENARIO VER="105">

<TXSPN>皆様、お久しぶりです。<BR>

わかばかおる＠帰りの会２です。<PG>

このたびは、「里奈の小さな丘」をご購入いただき、ありがとうございます。<PG>

{MESOFF "0"}

{WAIT "500" "CLICK" "SKIP"}

前作「麻衣のつぼみ」から4年ぶりの作品となってしまいました。<PG>

前作の後、お世話になっている先輩の会社を手伝うことになり、空き時間にコツコツと制作をしていたら4年たっていました。<PG>

{MESOFF "0"}

{WAIT "500" "CLICK" "SKIP"}

今作もPlaneRunnerによる変形動画を中心とした作品になりました。<PG>

ちょっと新しい試みで、モーフィングに挑戦してみました。<BR>

気づいていただけたでしょうか？<PG>

具体的には、<PG>

lmlsb edit 00000001.lsb 180

Textins [<TXSPN>皆様、お久しぶりです。<BR>]:<TXSPN>Everyone, long time no see.<BR>
Textins [わかばかおる＠帰りの会２です。<PG>]:It's the meeting for the return of wakabaka.<PG>
Textins [このたびは、「里奈の小さな丘」をご購入いただき、ありがとうございます。<PG>]:Thank you for purchasing "Little Hill of Rina".<PG>

If this is possible, there is a chance to leave the rest untouched.

It seems every lsb has only one TXSPD / TXSPF / TXSPN / TXSPS, breaks the line with BR and you have to click inside the game for every PG.

There might be other coding between the text, but it should be traceable for editing.

pmrowla commented 5 years ago

As far as inline script editing goes, it's certainly possible to do with the pylivemaker API, but I think that's a bit beyond the scope of the lmlsb edit CLI tool. Parsing and (re-)compiling a scenario script requires a lot of context specific information which is why the current CLI tools only support extracting and replacing complete scripts.

Basically, all of the existing CLI tools I wrote were just proof-of-concept examples, and were not meant to be particularly useful for patching an entire game.

Also, the {STOPMOVIE} issue (which should be fixed) was a parser bug that was not actually specific to {STOPMOVIE} events, and would have needed to be addressed eventually. I'm actually surprised that particular bug hadn't affected more scripts up until now.

LioMajor commented 5 years ago

Even it it's just POC, you did a very good job on it !!!

Most problems were fixable even if i there was the need to hex edit such as region codec trouble for unsupported characters in windows default fonts (ex. ① hex 87 40).

So far until now, i don't know of any other problems i had using your and the old tools except the wish there would be a way to upgrade older lm2 to lm3 scrips for the purpose to disable letter spacing (lm2 doesn't support it and you can't replace windows default fonts).

I thank you for the time you spend on this and if there is a way to donate, i will do.

KimchiTea commented 4 years ago

I'm trying to edit a game's menus from Japanese to English, which are located in ノベルシステム/■初期化.lsb as strings: 69: Calc StringToArray("文字を消す,シナリオ回想,読んだ文章を飛ばす,自動テキスト送り,セーブ,ロード,オプション,読んだ文章を自動的に飛ばす,テキスト速度...,自動テキスト送り時間設定...,フォント選択...,サウンドを再生する,音量調節...,BGM,効果音,セリフ,MIDI出力ポート選択...,フルスクリーン,ディスプレイモード...,ゲーム終了,タイトル画面に戻る,終了", システムメニュー項目名, ",") 70: Calc StringToArray("文字を消す,シナリオ回想,読んだ文章を飛ばす,自動テキスト送り,セーブ,ロード,オプション,オプション_読んだ文章を自動的に飛ばす,オプション_テキスト速度,オプション_自動テキスト送り時間設定,オプション_フォント選択,オプション_サウンドを再生する,オプション_音量調節,オプション_音量調節_BGM,オプション_音量調節_効果音,オプション_音量調節_セリフ,オプション_MIDI出力ポート選択,オプション_フルスクリーン,オプション_ディスプレイモード,ゲーム終了,ゲーム終了_タイトル画面に戻る,ゲーム終了_終了", システムメニュー項目値, ",")

Is it possible to make these editable with lmlsb edit? Thanks.

pmrowla commented 4 years ago

It should be doable, I'll take a look into it whenever I get a chance

LioMajor commented 4 years ago

@KimchiTea until it gets added as feature, it's possible for menu, savescreen and game choices to do it manually.

Example: 文字を消す

Open ■初期化.lsb in HxD or your prefered hex editor
Convert 文字を消す to hex (I'm using Translation Aggregator for this - sjis to hex)
Search for it in hex (95B68E9A82F08FC182B7)
You have to change it to whatever you want it (you might need to enlarge or shorten ■初期化.lsb)

Here comes now the trick, the lsb has a value for each string array to mark how long (length in hex) it is inside the file. Ex. in aboves picture is 3A 01 (always aftert 04).

Change the length to make it work again Ex. new length is 010A (reversed in file)

Hint: You have to use a new savegame, using ' old ' savegames might not show your changes as it partly stores game text plus everything recently before saving!

Make sure your edit works when starting a new game

KimchiTea commented 4 years ago

Whoa, thanks! I think I edited the hex values correctly, but now for some reason I can't patch the game. Here's the error I get for $ lmpatch MyGame.exe ノベルシステム/■初期化.lsb:

Generating new archive contents... Cleaning up temporary files... Traceback (most recent call last): File "/usr/local/bin/lmpatch", line 11, in <module> load_entry_point('pylivemaker==0.1.2.dev0', 'console_scripts', 'lmpatch')() File "/home/amy2/.local/lib/python3.6/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/home/amy2/.local/lib/python3.6/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/home/amy2/.local/lib/python3.6/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/amy2/.local/lib/python3.6/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/pylivemaker-0.1.2.dev0-py3.6.egg/livemaker/patch.py", line 147, in lmpatch raise e File "/usr/local/lib/python3.6/dist-packages/pylivemaker-0.1.2.dev0-py3.6.egg/livemaker/patch.py", line 121, in lmpatch new_lm.write(lsb_path, compress_type=info.compress_type, unk1=info.unk1) File "/usr/local/lib/python3.6/dist-packages/pylivemaker-0.1.2.dev0-py3.6.egg/livemaker/archive.py", line 782, in write with open(name, 'rb') as f: FileNotFoundError: [Errno 2] No such file or directory: 'ノベルシステム\\■初期化.lsb'

@LioMajor were you able to patch? I don't know if this is related to the file editing or a separate issue that should go in a new thread.

pmrowla commented 4 years ago

StringToArray support added in 8d1bfa97dbe89b5ad094e952ee4a0cd26f580446

py:pylivemaker ❯ lmlsb edit ノベルシステム/■初期化.lsb 65                               ⏎
65: Calc StringToArray("文字を消す,シナリオ回想,読んだ文章を飛ばす,自動テキスト送り,セーブ,ロード,オプション,読んだ文章を自動的に飛ばす,テキスト速度...,自動テキスト送り時間設定...,フォント選択.
..,サウンドを再生する,音量調節...,BGM,効果音,セリフ,フルスクリーン,ゲーム終了,タイトル画面に戻る,終了", システムメニュー項目名, ",")

Editing Calc expression
  StringToArray("文字を消す,シナリオ回想,読んだ文章を飛ばす,自動テキスト送り,セーブ,ロード,オプション,読んだ文章を自動的に飛ばす,テキスト速度...,自動テキスト送り時間設定...,フォント選択...,サウ
ンドを再生する,音量調節...,BGM,効果音,セリフ,フルスクリーン,ゲーム終了,タイトル画面に戻る,終了", システムメニュー項目名, ",")
  Array variable [システムメニュー項目名]:
  Array entry 0 ["文字を消す"]: "Erase characters"
  Array entry 1 ["シナリオ回想"]:
  Array entry 2 ["読んだ文章を飛ばす"]:
  Array entry 3 ["自動テキスト送り"]: "Automatic text feed"
  Array entry 4 ["セーブ"]:
  Array entry 5 ["ロード"]:
  Array entry 6 ["オプション"]:
  Array entry 7 ["読んだ文章を自動的に飛ばす"]: "Auto skip read sentences"
  Array entry 8 ["テキスト速度..."]:
  Array entry 9 ["自動テキスト送り時間設定..."]:
  Array entry 10 ["フォント選択..."]: "Font selection..."
...
  Array entry 19 ["終了"]:
  Array entry separator [","]:
Backing up original LSB.
Wrote new LSB

py:pylivemaker ❯ lmlsb dump ノベルシステム/■初期化.lsb
...
65: Calc StringToArray("Erase characters,シナリオ回想,読んだ文章を飛ばす,Automatic text feed,セーブ,ロード,オプション,Auto skip read sentences,テキスト速度...,自動テキスト送り時間設定...,Font
 selection...,サウンドを再生する,音量調節...,BGM,効果音,セリフ,フルスクリーン,ゲーム終了,タイトル画面に戻る,終了", システムメニュー項目名, ",")
...

things to note for @KimchiTea :

I have not tested translated menus in a patched game, and I'm not sure what will happen to the menu display for translations that are longer than the original string
You definitely do not want to edit/translate the variable name (first edit prompt) for system scripts
You probably do not want to change the separator string (final prompt, usually a comma) unless you really need to have a comma in one of your entry translations

LioMajor commented 4 years ago

@pmrowla thanks for your update

@KimchiTea

I haven't tested pmrowla's new update and don't know if the rest will work too.

I have patched by hex already a few games and it works fine when the length is set to the new correct value even to put them back to the game.

In system files, you have to double check what you change to not break it.

To test before patching the lsb back to the exe or dat file, you can copy the lsb to the correct path and it will be read first ignoring the original.

Example of files you might want to edit: game.exe (fullscreen / quit is inside the exe file on top) 00000001.lsb (translated file) and so on... メッセージボックス作成.lsb (fixed letter spacing for ingame text) not for older livemaker version! game.ext (if vff part is not inside exe) game.dat (if vff part is not inside exe) game.001 (if vff part is not inside exe) ノベルシステム\■初期化.lsb (game menu) ノベルシステム\シナリオ回想\初期化.lsb (scrollback + history letter spacing) not for older livemaker version! ノベルシステム\メッセージボックス\プレビューメニュー処理.lsb (scrollback + history letter spacing) not for older livemaker version! ノベルシステム\■オプション_自動送り時間.lsb (automatic text timing s and 10s) ノベルシステム\■オプション_テキスト速度.lsb (Text speed) ノベルシステム\システムメニュー\オプション待ち時間スライダー変化時.lsb (s) ノベルシステム\システムメニュー\オプション選択時.lsb (10s) ノベルシステム\システムメニュー\終了選択時.lsb (Return and quit question) ノベルシステム\システムメニュー\セーブ.lsb (save name suggestion + save position) ノベルシステム\システムメニュー\セーブメモ.lsb (cancel) ノベルシステム\システムメニュー\ロード.lsb (load position) ノベルシステム\メッセージダイアログ\■ダイアログ表示.lsb (cancel) ノベルシステム\メッセージボックス\文字列入力.lsb (cancel)

KimchiTea commented 4 years ago

Well, editing the strings with lmlsb worked fine, but I think I broke something when I updated because now lmpatch doesn't work at all... Here's the error message I got.

Traceback (most recent call last): File "/usr/local/bin/lmpatch", line 11, in <module> load_entry_point('pylivemaker==0.1.3.dev0', 'console_scripts', 'lmpatch')() File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 480, in load_entry_point return get_distribution(dist).load_entry_point(group, name) File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2693, in load_entry_point return ep.load() File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2324, in load return self.resolve() File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2330, in resolve module = __import__(self.module_name, fromlist=['__name__'], level=0) File "/usr/local/lib/python3.6/dist-packages/pylivemaker-0.1.3.dev0-py3.6.egg/livemaker/patch.py", line 31, in <module> from livemaker import LMArchive, LMCompressType ImportError: cannot import name 'LMCompressType'

Sorry, I know this is kind of off-topic from the original issue, but I could use some help. Thanks.

pmrowla commented 4 years ago

@KimchiTea looks like I broke that, should be fixed now with the latest commit (0bf308c5adff5ef57e8612e1850df6009d4c41d9)

KimchiTea commented 4 years ago

@pmrowla Thanks very much! And I realized what my previous patching issue was; not sure if it's pylivemaker specifically or Ubuntu in general, but something doesn't seem to like Japanese characters in folder names. However, a workaround is to open a terminal window in the home folder rather than the folder you're working in and copy/paste the full file paths.

EDIT: Actually, never mind, that doesn't work; it says it patched successfully but when I extract the patched .exe, it still has the old lsb files.

H0R1Z0N commented 4 years ago

@LioMajor I tried editing the right click menu of anaheim's item name array and patching it back to the game's exe but the exe don't seem to load the file, so i tried to also make the folder who hold it in the decompressed state and putting it there, no dice. Everything else seem to work, wondering if it's just Wine not liking JP name even using my shift-jis patch (Which's from Fedora which you can get at my website brokendragontranslation.com). Every array was translated but i left the holding name of the array as is. Used lmlsb edit command.

Edit : For weird reason, it's working now... Probably a cache issue.

KimchiTea commented 4 years ago

@H0R1Z0N I haven't been able to get Japanese LiveMaker games to work in Wine. Games that were originally published in English work, but not games published in Japanese even if they're English patched. They run initially, but clicking start either does nothing or leads to a black screen. Tried your SJIS patch with no change. So if you can let me know how you got it to work, I'd be very grateful.

H0R1Z0N commented 4 years ago

Which distro are you using ? Well, after downloading the local patch from freebsd (fedora pull it from freebsd), you need to install it has a new local like the guide say, then, you need to copy all the windows font into the wine font directory. Then, navigate in the terminal to your game's exe folder, then run "LC_ALL=ja_JP.sjis wine XYZ.EXE" or on lutris if your lutris's was fixed from the breaking update last week with the environement setting with one column being LC_ALL and the other ja_JP.sjis

H0R1Z0N commented 4 years ago

You also need to force the local with the argument it show in the error if you get the error about it not being ansi or ascii.

KimchiTea commented 4 years ago

I'm running Ubuntu 18.04.4. I'm not read up on what the difference between distros is but that's probably the issue. I'm not sure what you mean by force the locale. I guess it didn't install correctly, as I got an error "ja_JP.SHIFT_JIS...[warning] character map `SHIFT_JIS' is not ASCII compatible, locale not ISO C compliant [--no-warnings=ascii]", but if I try to run "sudo locale-gen --no-warnings=ascii" I get an error that "--no-warnings" isn't a valid option.

Clicking start in the menu of a LM game gives a "Unhandled exception 0x0eedfade" error.

No big deal if it doesn't work, I usually just run LM games in a WinXP virtual machine, but I'll link your SJIS patch in my patch release notes in case it helps anyone else.

H0R1Z0N commented 4 years ago

The command need to be issued on localedef -i ja_JP -f SHIFT_JIS ./ja_JP.sjis --no-warning=ascii

Also, it's not gonna be easy with a very old ubuntu like the LTS one since i used latest version based on latest fedora while your version of glibc might be way older. When i made the patch the first time, glibc got update 2 day after, then it was never updated since.

H0R1Z0N commented 4 years ago

Be sure that the local.gen got the declaration correctly.

KimchiTea commented 4 years ago

Still getting the same error. I'm not going to worry about it for now, might try to work it out later. Thanks for your help, though!

H0R1Z0N commented 4 years ago

So, to resume, i was able to do most of the stuff with hexedit, there's still a few thing left to do... But, here's the stuff i have some issue atm.

The backlog dialogue box for the full backlog from the right click was perfectly fine to make it singlewide. For what ever reason, the scroll up backlog's not working in singlewide when i edited that file mentioned above. Still going to look more about it later.

My second issue's the fact that the quit game dialogue box call "@Title" and i cannot seem to find that one in any script, nor the exe... I am still digging around the file to find it.

H0R1Z0N commented 4 years ago

I forgot to mention, the reason i want to modify the @Title's for having the title in the window in english and the same with the Quit "Insert game name" now? dialogue box.

H0R1Z0N commented 4 years ago

NVM for the issue with backlog dialogue box, i am an idiot who cannot read japanese and confused a folder with a folder with similar name and hence why the game was not loading the file and it seem the patch was not working or and i was totally thinking i had the wrong file...

pmrowla commented 4 years ago

@Title is a a read-only system var, I think it's the actual window/application name set in the .exe, it's not set anywhere in LSB files.

basically everything in https://pylivemaker.readthedocs.io/en/latest/livenovel/basic.html?highlight=%40title#system-variables comes from making windows system calls and not from LM scripts

H0R1Z0N commented 4 years ago

Was kinda expecting it since the Quit and Fullscren button was in the exe.

Do you think it's doable to translate the position menu on the top left when you right click for the menu ?

pmrowla commented 4 years ago

I think it should be possible to hexedit the string in the exe, I can take a look at the agls exe in ghidra, but automating it for any game in pylm might be a bit complicated.

pmrowla commented 4 years ago

@H0R1Z0N to translate the top left context menu, you have to hexedit bytes from the actual exe:

offset    contents
0x1A2D00: 0x04 (4-byte int)
0x1A2D04: 終了 (cp932 str, 8-bytes max including null)

0x1A2D10: 0x08 (4-byte int)
0x1A2D14: 画面切替 (cp932 str, 12-bytes max including null)

ints are little endian and store the length of the string in bytes (not including null terminator), the translated strings have to fit into the space before the next set of "FF FF FF FF" bytes. I tested using "Quit" (len 4) and "Fullscreen" (len 10)

(this doesnt translate the "are you sure you want to quit" popup but is probably good enough?)

H0R1Z0N commented 4 years ago

I already did those, i am currently digging on those game file since i found nothing on ghidra either nor with the hexeditor. I noticed there was way more special file i had not touch from the extraction. If i find it, i will be sure to report back.

H0R1Z0N commented 4 years ago

I think i found it in live.lpb https://cdn.discordapp.com/attachments/692957773855588382/706439731545899035/unknown.png

pmrowla commented 4 years ago

It's also in INSTALL.DAT

H0R1Z0N commented 4 years ago

At this point, it could be the name of the directory too. I have a feeling the Install.dat's more for uninstalling the game once someone's done, so it delete the directory, i might be wrong too.

pmrowla commented 4 years ago

I think it's probably live.lpb, since that's the file they use in TProject.Create

H0R1Z0N commented 4 years ago

https://cdn.discordapp.com/attachments/692957773855588382/706442309150965771/unknown.png It was indeed live.lpb

H0R1Z0N commented 4 years ago

https://cdn.discordapp.com/attachments/692957773855588382/706442826455449640/unknown.png It also fixed the close game dialogue

H0R1Z0N commented 4 years ago

Dang Peter, you made the lmlsb batchinsert way faster, it's blazing fast now on a very slow folder, instead of taking 20min to compile that folder, it's now done in less than 3sec. You are awesome.

pmrowla commented 4 years ago

that improvement was mostly thanks to @Stefan311

there's also issue to support editing lpb directly now: #39

LioMajor commented 4 years ago

The lpb's title is very easy to hex edit.

It's right at the start and has (blue marked) a hex value for the length:

pmrowla commented 4 years ago

lmlpb tool is now available in master, has probe and edit commands that work the same way as lmlsb

@LioMajor have you tried playing around with anything else in the lpb before? I noticed there's a default font setting in live.lpb.

LioMajor commented 4 years ago

@pmrowla yes, when i tried to prevent letterspacing i checked if i can just use a different font.

The default only changes the possible fonts and it didn't work well to use a non listed.

Without save.dat it will be the default font (ＭＳゴシック) until you change it. To edit live.lpb 's font does only change the default.

pmrowla / pylivemaker

lmlsb edit improvements #11