search

Pyrdacor / Ambermoon.net

Ambermoon rewrite in C#
GNU General Public License v3.0
417 stars 20 forks source link

[Feature] Add game data writers #53

Open Pyrdacor opened 3 years ago

Pyrdacor commented 3 years ago

Those writers should allow to save game objects back to files. Useful for savegame writing, game data patching or modding.

Writers needed for:

Maybe also add palettes writer later? Maybe also allow graphic writing later?

a1exh commented 3 years ago

How are the writers used?

Pyrdacor commented 3 years ago

At the moment you need to program in C#. So I can use them to change data. I need a game version which I should patch and a list of patches to perform.

Pyrdacor commented 3 years ago

Slow is ok. I have to implement battles, too. I can read german so you can focus on the english version. Thanks for the links.

I also prefer a file by file approach. If you don't know in which file a bug may be located, feel free to ask. I am very familiar to the internal data structures and their relations already.

a1exh commented 3 years ago

One bug was described as a typo "YELLOW CARD SPHER" changed to "YELLOW CARD SPHERE" but don't know what file this is in. I didn't see it in any *_texts.amb file. It is possibly an item? So in Save.00/Chest_data.amb ?

Pyrdacor commented 3 years ago

It is an item but they are stored in AM2_CPU and AM2_Blit.

Recently I found another issue. Don't know if this is fixed. When you leave the airship on forest moon you can trigger the witch broom message twice. We should change that by a map event condition.

a1exh commented 3 years ago

Thanks, that explains it, the patches that were made to AM2_CPU and AM2_BLIT by Hexaae in approx. 2001 (including 2-3 other fixes) I have unfortunately lost at sometime in the past. I think you discovered they were packed using imploder. I am able to depack using the Amiga tool xfddecrunch, the game still runs depacked, I can find the strings but I don't understand the data structure enough to insert bytes into strings without breaking something (e.g. string length).

a1exh commented 3 years ago

Recently I found another issue. Don't know if this is fixed. When you leave the airship on forest moon you can trigger the witch broom message twice. We should change that by a map event condition.

No-one has reported this bug previously. A save game near by and technical description would be good.

Pyrdacor commented 3 years ago

The strings in AM2_CPU are a bit strange but luckily the item names have fix lengths. I can help editing them. You can decompress and compress the AM2_CPU / AM2_Blit with imploder-4 in WinUAE.

Pyrdacor commented 3 years ago

Recently I found another issue. Don't know if this is fixed. When you leave the airship on forest moon you can trigger the witch broom message twice. We should change that by a map event condition.

No-one has reported this bug previously. A save game near by and technical description would be good.

I can provide a savegame tomorrow. When you leave the airship on forest moon the first time with the witch's broom in inventory and then stumble upon a nearby tile an event is triggered with a text popup. The text says that a demon appears and crushes the broom. Then the broom is destroyed. But when you go to another tile nearby, the same popup is triggered again. This should not happen.

a1exh commented 3 years ago

One of the more aesthetic issues in v1.07 are the translations of the stone head limericks (rhymes). In German they all rhyme and are fun but whoever translated them did the equivalent of Google translate and didn't attempt to re-create the style of the German versions. Thinking up new text is relatively easy. Getting them into the files without making a mistake is harder. This is where the data-writers may help.

e.g. 2Map_texts.amb - 103.amb

The German version is this :

DEINE STIMME KENN ICH NICHT, DU KANNST HIER NICHTS ERREICHEN! DOCH SAGST DU MIR DAS RICHTGE WORT, DANN LASS ICH MICH ERWEICHEN.

DICH DRÄNGT DER DURST? DU SUCHST DEN WEIN? WOHLAN DENN, GEH DEN GANG HINEIN!

But the translation falls so flat

I DO NOT KNOW YOUR VOICE, YOU CANNOT ACHIEVE ANYTHING HERE! BUT IF YOU SAY THE RIGHT THING, THEN I SHALL LET YOU GO.

"ARE YOU THIRSTY? WOULD YOU LIKE SOME WINE? PERHAPS THEN GO INTO THE PASSAGEWAY!"

A German fan pointed out all these translation issues even suggesting some better translations :

I DO NOT RECOGNISE YOUR VOICE, WHICH LEAVES ME NO OTHER CHOICE! I WILL ONLY LET YOU THROUGH, IF YOU ANSWER ME THIS CLUE!

YOU SUFFER REAL THIRST? WOULD YOU LIKE SOME WINE? LOOK IN THE CORNER FIRST, DON'T WASTE ANY TIME!

However modifications like this manually without any checking, especially with mark-up for bold etc. is why they have not been fixed yet.

a1exh commented 3 years ago

When you leave the airship on forest moon the first time with the witch's broom in inventory and then stumble upon a nearby tile an event is triggered with a text popup. The text says that a demon appears and crushes the broom. Then the broom is destroyed.

This is interesting, I don't remember this. Daniel Schulz reports for v1.05 that you can smuggle a broom onto Morag's moons using two spells WORD OF THE MARK and WORD OF RETURN.

Hexenbesen nach Morag schmuggeln Auf dem Wüstenmond wird ein eventuell mitgeführter Hexenbesen per Event zerstört. Über die Zauber "Wort des Markierens" und "Wort der Rückkehr" lässt sich das Triggern dieses Events umgehen, und damit der Besen als Transportmittel auf dem Mond benutzen.

Pyrdacor commented 3 years ago

The data writer will have no problem adding or editing map texts like the riddles.

About the broom event. I think the best solution would be to trigger the event right when leaving the airship.

Pyrdacor commented 3 years ago

Tomorrow I will give you 3 things:

You can then edit all map texts to your liking.

Pyrdacor commented 3 years ago

Maybe better?

I DO NOT KNOW YOUR VOICE,
WHICH LEAVES ME WITH NO OTHER CHOICE!
I WILL ONLY LET YOU THROUGH,
IF YOU ANSWER ME THIS CLUE!

YOU LOOK FOR WINE?
YOUR THROAT IS DRY?
THE ANSWER WAS FINE.
YOU CAN PASS BY.
Pyrdacor commented 3 years ago

@a1exh Here is the savegame on the forest moon. Go outside the airship. The text should popup. Then go 1 tile right or left. The text should popup again. Save.07.zip

Pyrdacor commented 3 years ago

@a1exh Confirmed. In english 1.07 Reg is still bugged. He is not marked as boss. You can cast Fear/Panic on him and he would flee with the items you actually need.

a1exh commented 3 years ago

Please can you confirm the technical information for the fix?

I extract sub-file 012.amb in monster_char_data.amb (the one which contains the string "REG HILL-GIANT")

Using the information from here (thank you very much) :

https://github.com/Pyrdacor/Ambermoon/blob/master/FileSpecs/Characters.md

Byte offset 0x0012 (Monster Flags) needs bit 2 (Boss) set to 1?

In v1.07 "Monster Flags" for REG HILL-GIANT is 0x02 and I change it to 0x06?

Pyrdacor commented 3 years ago

The sub-file should be 018.amb (sub-files are counted starting at 1).

I would change the value to 0x04 (only bit 2) because bit 1 seems to be for demon monsters only.

I also recommend to set byte 0x13 from value 0x01 to 0x00. I don't know what this bit means but only the guard demon has it set and it would be strange if Reg has this bit as well.

The rest is fine. ;)

Pyrdacor commented 3 years ago

The tool to export and import texts is finished and is working on my end: https://github.com/Pyrdacor/Ambermoon/releases/download/v1.0.0/AmbermoonTools-Windows.zip

Usage:

I used it like this to edit some texts in grandfather's house:

  1. AmbermoonTextImport.exe -e Amberfiles 2Map_texts.amb MyExportedTexts
  2. Edit the text files (in this case they are in "MyExportedTexts\2Map_texts.amb\258")
  3. AmbermoonTextImport.exe -i Amberfiles 2Map_texts.amb MyExportedTexts

After restarting Ambermoon the texts should be updated. Let me know how it went. ;)

Quick note: The text files are stored and loaded in UTF-8 format. So don't mess up the encoding. ;) But for english texts that should not matter at all as ASCII is equivalent to UTF-8 for all english characters.

a1exh commented 3 years ago

The sub-file should be 018.amb (sub-files are counted starting at 1).

I am using the Amiga Ambermoon tools by Metallic. They label the files in HEX so 018.amb is the same as (0x)012.amb. All references to sub-files in my spreadsheet are in hex

a1exh commented 3 years ago

If you don't know in which file a bug may be located, feel free to ask. I am very familiar to the internal data structures and their relations already.

In the bandits cellar, there are a few rooms where you have to push a button or pull a lever to open passages. One of those walls has a second, non-functional, button. It's similar to the stairs in Sansrie's temple and Luminor's Tower which have the wrong gfx and show up not down and vice versa. These are caused by an incorrect map entry (e.g. the wall should have only one button, and the wall tile with the second button should be a regular wall tile).

This is devilishly hard to get correct using a HEX editor. It would be good to be able to edit the MAP data in a more efficient manor or at least be able to test the dungeon to change mistakes when it comes to the MAP data writer.

Pyrdacor commented 3 years ago

The sub-file should be 018.amb (sub-files are counted starting at 1).

I am using the Amiga Ambermoon tools by Metallic. They label the files in HEX so 018.amb is the same as (0x)012.amb. All references to sub-files in my spreadsheet are in hex

Ah ok makes sense. Then this is ok.

Pyrdacor commented 3 years ago

If you don't know in which file a bug may be located, feel free to ask. I am very familiar to the internal data structures and their relations already.

In the bandits cellar, there are a few rooms where you have to push a button or pull a lever to open passages. One of those walls has a second, non-functional, button. It's similar to the stairs in Sansrie's temple and Luminor's Tower which have the wrong gfx and show up not down and vice versa. These are caused by an incorrect map entry (e.g. the wall should have only one button, and the wall tile with the second button should be a regular wall tile).

This is devilishly hard to get correct using a HEX editor. It would be good to be able to edit the MAP data in a more efficient manor or at least be able to test the dungeon to change mistakes when it comes to the MAP data writer.

I agree. But it is also hard to create a tool to edit such stuff in an easy way. Maybe you have some ideas. Command line tools can be created rather fast. I can also print all map properties by name (like "Tile[1,1]" or "MusicIndex", etc).

Pyrdacor commented 3 years ago

Did you try the text importer?

a1exh commented 3 years ago

ASCII representation of the dungeon levels using different characters for the tiles?

I have not yet tried the text importer. I will try it soon as I have several files I hand edited and changed string lengths and I want to see if they are the same.

Pyrdacor commented 3 years ago

What do you mean by ASCII representation of dungeon levels?

a1exh commented 3 years ago

Did you never play text based MUDs? (I am showing my age here). I was thinking about printing a 2D layout to the console window with different characters for wall textures etc.

Thinking about it , it is a pointless idea, the better idea would be to be able to edit the map data somehow, perhaps as text files, write them back to the data files, launch the game and interact.

Perhaps this could be done in conjunction with the Savegame writer? If you can quickly and easily place your party near to the thing you're editing?

Pyrdacor commented 3 years ago

Sorry I misread your comment. Of course I know what you mean. But wall and objects indices can be greater than 100. We will ran out of printable characters quickly.

Maybe I can export maps as xml or json.

a1exh commented 3 years ago

Another curiosity that maybe the data writers can answer. v1.07 2Map_texts.amb sub-file 1C6.amb and 1B7.amb "HANGAR ON MORAG" are ALMOST the same file. The two files must have existed in the German version and been given to the translation company as two files, they have been translated separately and as a result in v1.07 each one contains slightly different English translations for the same text. I am curious why there are two sub-files the same? I wonder if they were identical in the German version? Sub-file 1C6.amb is definitely used in v1.07 as it has a patch already. Perhaps 1B7.amb is never used? Or one is used for the first visit and the other for return visits?

Pyrdacor commented 3 years ago

You are right. The two files are identical in german. In english the translation is a bit different for 3 of the 5 texts. They were translated separately but the texts have the same meaning in english too.

I will check how they are used.

Pyrdacor commented 3 years ago

Ok map 0x1C6 can be accessed from the airship on Morag and from the town of S'Angrila. Map 0x1B7 can only be accessed from the airship on Morag.

There is a condition which checks for global variable bit 222. If you leave the airship and the variable bit is 0, map 0x1C6 is entered, otherwise map 0x1B7. I will check where variable 222 is set and how both maps differ.

Pyrdacor commented 3 years ago

When talking to S'Riel he will tell you about the plan of the Morag people and then teleports you to his tower. Then the global variable 222 is set to 0 so later when leaving the airship on Morag you will enter map 0x1C6.

Background: When you first enter morag via airship you will enter map 0x1B7. It contains an event which teleports you to the prison cell where S'Riel is dying. After talking to him he will teleport you to his tower. After that you will never enter map 0x1B7 again but map 0x1C6 instead which does not contain the teleport event to the prison.

The different texts are just for observing some things in the hangar and are slightly different.

a1exh commented 3 years ago

It is not just these two files, it is also 1C7.amb and 1B8.amb S'ANGRILA PRISON (makes sense as presumably if you can return to this map location twice it also needs to be modified not to contain previous content)

Daniel Schulz has some notes about a bug connected to this :

HANGAR AUF MORAG und GEFÄNGNIS - S'ANGRILA sind nicht auf dem Wüstenmond Die Karten "HANGAR AUF MORAG" und "GEFÄNGNIS - S'ANGRILA" sind als "auf Lyramion" gekennzeichnet, sind aber eigentlich auf dem Wüstenmond. Dadurch lassen sich dort die Zaubersprüche WORT DES MARKIERENS und WORT DER RÜCKKEHR wirken, was den oben beschriebenen Trick ermöglicht. (Das gilt allerdings nur für das erstmalige Betreten des Mondes; nach dem Event mit dem sterbenden Moraner werden fortan 2 Duplikate dieser Karten verwendet, die korrekt als "auf Morag" gekennzeichnet sind.)

HANGAR ON MORAG and S'ANGRILA PRISON are not on the desert moon. The map locations "HANGAR ON MORAG" and "S'ANGRILA PRISON" are marked as "on Lyramion", but are actually on the desert moon MORAG. This allows the WORD OF MARKING and WORD OF RETURN spells to work there, which enables the trick described above. (This only applies to the first time you step onto the moon; after the event with the dying Moraner, 2 duplicates of these map locations will be used, which are correctly marked as "on Morag".)

Pyrdacor commented 3 years ago

We can just change the world of these 2 maps. This will fix the bug. The world byte is documented in the other repo.

But I wonder why they used 2 maps. Events can be deactivated. This is often used. Don't know why 2 maps are necessary.

a1exh commented 3 years ago

Any chance you could add a switch to the data writers so they wrote filenames with HEX numbering. It would make my life a bit easier

a1exh commented 3 years ago

I tried the extract and insert to modify the file. It worked well. A few minor notes.

It has removed lots of $20 characters from within the sub-file text blocks. (Whitespace?) Is this deliberate? It does make RAW diff more difficult.

It didn't LOB pack the sub-files. Was that to be expected?

a1exh commented 3 years ago

More importantly the files can be loaded and extracted by the existing Amiga tools. All the fixes I am attempting, unfortunately I don't have save games anywhere near the affected areas so I cannot test in-game.

A switch to preserve whitespace would be good, so the files could be easily diffed against the originals.

Pyrdacor commented 3 years ago

I will add switches for hex names and whitespace preserving.

About the LOB compression. The text files use the AMNP format. This is an archive where all files are automatically JH encoded and optionally LOB compressed. As the LOB compression is optional I only use LOB if the file size would be smaller. So my tools will always produce the smallest file sizes possible. If you want, I could also add a switch for forcing LOB compression on AMNP but I don't know if this would be desirable.

a1exh commented 3 years ago

No that is fine, these were small files and to be honest I didn't check to see if the originals were LOB packed

Pyrdacor commented 3 years ago

Indeed the originals are. But I guess they just used LOB compression manually on those files without checking if it really is of any use on a file by file basis.

But as I'm writing this I checked and found that my produced files are much bigger. So I might do a mistake here. I will check.

Pyrdacor commented 3 years ago

Which format for filenames do you prefer? Just 1C0.txt? Or $1C0.txt or 0x1C0.txt? Do you want the directory names (which represent the map index) also in hex?

a1exh commented 3 years ago

The important bit < for me > is the sub-file directory name. MyExportedTexts/1Map_texts.amb/1C0/000.txt

Pyrdacor commented 3 years ago

Ok I added all the features but I will dig into the larger file size a bit before releasing.

Pyrdacor commented 3 years ago

The archive format usage is a bit strange.

AMNP uses JH and then optionally compresses with LOB. AMPC doesn't use JH but always LOB.

In the first case (which is used for the text files) the LOB compression isn't very good cause JH leaves very random byte data which is not compressable very good. So I don't understand why AMNP doesn't use the other way around: first LOB then JH. Moreover I don't get why the original data files were compressed very well with AMNP.

If I use my own LOB algorithm and use AMPC (LOB only) the file size of 2Map_texts.amb is even smaller than that of the original. But with AMNP it is much larger which makes sense as JH + later LOB is bad. I guess there is something about the AMNP format I don't understand yet.

Pyrdacor commented 3 years ago

Ok I got it. AMNP is indeed first LOB compressed and then JH encoded. This makes much more sense. I will fix that. It worked cause the compression was always bad the other way around so it never used compression. :D

Pyrdacor commented 3 years ago

@a1exh I just released 1.0.2 of the tools. Can you try it please? https://github.com/Pyrdacor/Ambermoon/releases/download/v1.0.2/AmbermoonTools-Windows.zip

There are 3 new options: --preserve-whitespaces (or -p) --preserve-zero-bytes (or -z) --hex-subfile-names (or -h)

You can also use -pzh to use all 3 options in a shorter way. Of course you can also use -pz or any other combination as well. But do not mix it with -e or -i. So this is not allowed: -epz etc.

Examples:

AmbermoonTextImport.exe -e Amberfiles 2Map_texts.amb MyMapTexts -pzh AmbermoonTextImport.exe -i Amberfiles 2Map_texts.amb MyMapTexts -pzh

Note: Always use the same hex subfile name option for export and import.

The compression and AMNP writing should be fixed. The 2Map_texts.amb is even a bit smaller than in original.

I added the differentiation between whitespaces and zero-bytes to add more flexibility.

a1exh commented 3 years ago

Thanks for the quick fixes. -h works great.

The -p option doesn't seem quite right. The extracted files now include what I thought was the end of textblock marker (0x00) and in some cases whitespace (0x20) after the end of textblock marker I thought belonged to the start of the next textblock?

I wanted to see if I can use the tools to fix bugs where there is a missing end-of-textblock marker. There are at least 3 of these in v1.07 which cause the text parser to either crash, not display anything or display garbage.

2Map_texts.amb/105/003.txt 3Map_texts.amb/153/010.txt NPC_texts.amb/03A/007.txt

It has done a much better job than I thought it would do. I was expecting the textblock.txt file to contains two entries, and have to rename all the .txt files that come after by incrementing the number by one. But it has split them correctly. However with files containing non ASCII / UTF-8 characters now (e.g. 0x00) I don't think you can use a text editor to fix them.

Pyrdacor commented 3 years ago

The text files don't really use "end-of-text-markers" (I call them terminating zeros). Instead each text-file has a header which specifies the "total number of texts" and for each of them a "text length".

In original data some texts have spaces (0x20) or zero-bytes (0x00) at the end in any order (therefore you also see spaces behind 0-bytes). But they don't matter. This is why I removed them before as I don't need them in Ambermoon.net.

If you want to fix "end-of-text-markers" I think you really have to fix the "text length" entry at the beginning of the file. But it should be ok to just fix the text files with my tools. They will write the correct text length then.

Pyrdacor commented 3 years ago

So in short you should use the -p and -z options only for research (export only). When editing and re-importing the data just avoid these options. The text archive will be much smaller and nobody needs those spaces and zeros. Neither the original game nor Ambermoon.net. :)

Without the options the import will take care of the whitespace and zeros so you don't need to bother.

a1exh commented 3 years ago

Understood.

I was sure the Amiga code used both "text length" and "terminating zeros" and if they don't match, the text parser malfunctions.

The examples above I believe have correct "text length" but wrong "terminating zeros" and they make the Amiga text parser malfuction