Further reverse engineering

[WizzardMaker]

Figure out how

• [ ] *.pol
• [x] *.dat
• [ ] *.mcf
• [x] *.csv
• [x] *.raw
• [x] *.smk

are structured

[WizzardMaker]

There is already C++ (Or rather old C) code that can decode the *.csv files. They are encoded with an xtRLE method. http://karlchenofhell.org/kram/xtRLE_dec.c is working 100% Also here https://www.gog.com/forum/airline_tycoon_deluxe/editing_game_data_csv_files_to_customizes_cities_routes_etc

We have to add a smk decoder as well, for all the character animations in the rooms, a C# version is already in the open source StarCraft remake (https://github.com/kangaroo/scsharp/tree/master/SCSharp/SCSharp.Mpq.Smk), we should ask them if we can use and modify their code, would save us some work.

I don't know what the .dat files contain, but the misc folder has .res files which should contain all the dialog if my memory serves me right.

the *.mcf files contain the font data.

[WizzardMaker]

The *.dat files hold the airport gfx position data.

I managed to extract the information of those files. I created a script which reads the file and then creates a scene from it.

The file has the following structure: Header: 4 bytes (int) - Entries in file 4 bytes - ? often null

For each entry: 4 bytes (long) - file id (from brick.csv) 4 bytes (int) - x position 4 bytes (int) - y position 1 byte - other info, maybe room number 1 byte - null

There are still some artifacts, like misplaced tiles or multiple entries on the same position and its still missing a method to load animated entries.

Here is a little preview of the assembled airport from those files:

[Serphentas]

Amazing work ! The preview looks great, despite the few details here and there. I think that's a good basis to build upon.

I'll try to write a CSV handler/editor so that we can customize routes in-game.

[Serphentas]

Found info on xtRLE (but it's in German): http://www.karlchenofhell.org/kram.php?part=rhodan#xtRLE

[AWKIF2005]

I stumbled upon this project while trying desperately to find a way to decrypt the .csv files in ATD (specifically routen, planetyp, and city). First of all, I am beyond excited about the work you guys are doing to open ATD and bring it into the future. This is a classic game! I am not a coder or programmer, so most of what I see here goes completely over my head and I doubt I can be much help, but I will be following this project closely! Now to see if I can figure out how to use this script to open up and edit the .csv files...

Keep up the good work!

[Serphentas]

Thanks for the comment ! Much appreciated.

As for the script, it can only be used on relation.csv because other ".csv" files are either already CSV or something else (not xtRLE).

[WizzardMaker]

As for the script, it can only be used on relation.csv because other ".csv" files are either already CSV or something else (not xtRLE).

That's not 100% true. I successfully decoded all csv files with the aforementioned c file, doesn't mean your files are the same as mine, but they should be decryptable

I am porting the c file to C# atm. That should show us whether your files can be decoded or not.

xtRLE is also used in the Dialog files, but the current implementation of the script has some problems with the file, but that should be fixable.

[Serphentas]

That's what I get for en/data/*.csv:

brick.csv:    ISO-8859 text, with CRLF line terminators
builds.csv:   ASCII text, with CRLF line terminators
city.csv:     ISO-8859 text, with CRLF line terminators
clan.csv:     ISO-8859 text, with very long lines, with CRLF line terminators
names.csv:    ISO-8859 text, with CRLF line terminators
piloten.csv:  Non-ISO extended-ASCII text, with CRLF, NEL line terminators
planetyp.csv: ISO-8859 text, with very long lines, with CRLF line terminators
pnames.csv:   ISO-8859 text, with CRLF line terminators
relation.csv: data
routen.csv:   ASCII text, with CR line terminators
stdpaper.csv: ISO-8859 text, with CRLF line terminators

Do you have a similar layout ?

[WizzardMaker]

Well, seems like our versions differ a bit. My relation.csv and builds.csv are not encrypted and all other .csv files are encrypted with xtRLE

I always just look at the HEX content of the files to see whether they are encrypted or not

[AWKIF2005]

Perhaps the .csv files are in different states of coding depending on different versions or releases of the game. My Strategy First release of ATEvolution had the routen.csv file in regular text, so I was able to use Excel to easily mod the in-game route structure in ATE to my tastes, within some limits before the the game would start crashing on start up. I tried using the regular text routen.csv from ATE, manually added the new routes from ATDeluxe, following the same format, and then put that new routen.csv in the ATDeluxe data folder. It didn't work, the game crashes. I tried multiple variations of city spelling, etc, but no joy. Hence my desire to decode the actual routen.csv from ATD :) My version of ATDeluxe is a download from GOG games.

Interestingly, the builds.csv file is normal text in both versions, allowing easy modding of the "custom airplane" parts. I use this to create custom planes that mimic the specs of real aircraft that are not included in the original game (A330, A380, etc)

Hopefully part of the open ATD project will be to allow these types of mods more easily. Since I am a coding neanderthal, I would be happy to beta play the game for you guys, when it's ready! :)

[AWKIF2005]

WizzardMaker, sounds like you and I have the same version of the game as far as which .csv's are encrypted. Serphentas, your ATDeluxe routen.csv is already in normal text? Interesting!

[WizzardMaker]

Serphentas is using the Linux version (as far as I can see it) which was ported over, so they probably changed some parts of how they handle the files.

[AWKIF2005]

That probably explains it then.

[WizzardMaker]

Aaaand now we should be able to decrypt all the xtRLE files.

It seems like the c source from http://karlchenofhell.org/kram/xtRLE_dec.c had some problems with bigger files. I tested the following files with my C# decoder and all got decrypted successfully:

• misc/ein_ger.res
• misc/dlg_ger.res
• misc/std_ger.res
• data/brick.csv
• data/city.csv
• data/clan.csv
• data/Names.csv
• data/piloten.csv
• data/planetyp.csv
• data/pnames.CSV
• data/routen.CSV
• data/stdpapter.csv

I need to clean my code, but it should be committed in a couple of hours

[Serphentas]

Indeed, I run Linux and use the Linux version from Steam. Anyhow, just by looking at the first bytes of each file we can determine whether or not it is xtRLE-encoded and proceed with proper decoding accordingly.

@WizzardMaker this methodology could be implemented directly in ATDGameLoader.cs at import time, what do you think ?

[WizzardMaker]

@WizzardMaker this methodology could be implemented directly in ATDGameLoader.cs at import time, what do you think ?

Question is, do we need to pack these informations in our game package? There isn't really a need for us to have all of those files in the godot editor

We would need to create a system to handle most of those files correctly first I think.

The brick.csv file is only used by the airport loader at the moment, that doesn't need to be imported into godot.

The clan.csv file handles the IDs and animations of the people, that is also something that should be only loaded once like for the airport loadout, so that we can create the animations.

The other files are kind of cosmetic, like the names, planes or the routes. These are special, as they are the main target of modders. We should maybe include these in godot as special resources, so that modders can modify them.

[AWKIF2005]

@WizzardMaker The other files are kind of cosmetic, like the names, planes or the routes. These are special, as they are the main target of modders. We should maybe include these in godot as special resources, so that modders can modify them.

I like this idea. Plus it would be interesting to see how (or if) these modded files play with the open version you guys are working on.

[Serphentas]

In this case, I think it would be wise to create and maintain a Markdown file (instead of a Wiki page, just to keep everything within Git) where we detail how every file type is structured and what purpose it serves. This will not only serve as a reference for game data design but also help modders in their task.

We could already use information found by @WizzardMaker regarding graphics files.

[Serphentas]

Also, do we want our own format (or a standard format) that we will use later on for any game file or do we want to import game data as is ? In the latter case, modding will imply conversion to "classic" format and then reimporting to the game.

[WizzardMaker]

We should use our own format for all files we want to keep in the game. We can make our lives easier by using the Godot Resource class. With that we can save and load it like any other resource in godot. We enable in editor editing and we create an easy way to mod our files without studying the original game file structure

[AWKIF2005]

Okay, I admit defeat on this. I pride myself on being able to "figure it out" when it comes to computer stuff, but after many hours of trial and error and dozens of Google searches over the last few days I cannot figure out how to get the xtRLE code to decrypt the .csv files. Coding and programming is just not something I have needed to use up until now. If only I had gone to school to be a programmer instead of an airline pilot ;)

I know my problem is probably a little beyond the scope of the OpenATDeluxe project, but any hints or pointers would be much appreciated, from one Airline Tycoon fan to another. I am running Windows 10 with Visual Studio 2019 installed. I can't figure out how to point the script to the .csv files or vise versa. I tried compiling the c# script using the cmd prompt (csc), but it throws an "error CS0246: The type or namespace name 'file' could not be found (are you missing a using directive or an assembly reference?)" for each of the three instances of "(file fileIn)". The first time I tried compiling it there was also an error about the using godot reference, so I took that line out since I am trying to do this just in Windows.

Is there a way to define the path to the .csv file in the c# code, allowing it to find the file and run the decryption? Is compiling the .cs file to an .exe even necessary? Or am I thinking about this all wrong?

Thanks for any tips you guys might be able to provide to this noob over here!

[WizzardMaker]

You can't just run the code in visual studio, as we use functions from godot like the file handler

That's why your compile fails. You would need to either use godot or rewrite the code to use the .Net version of File, but that would need a bit more work

If you are currently only interested in the decrypted csv files of AT Deluxe, then write me an email and I will send them to you

[Serphentas]

As @WizzardMaker said, you need Godot unless you convert the script to native .NET. Otherwise, you can use this file (needs compilation) or simply this Python3 script.

[AWKIF2005]

@WizzardMaker emailing the decrypted .csv's to me would be incredible. Send em to shiz222@hotmail.com.

That makes sense. I didn't catch that the c# code was specific to godot. I did mess around with the project in godot a bit, but I couldn't get it pointed to at.exe properly, so I went back to trying the code in visual studio and cmd. I also tried the python script early on, but I don't think I got python set up properly on my 10 year old laptop (the only computer I currently have with me) :)

Folks like you guys make the internet awesome. Thank you. I will be following the project...

[Serphentas]

Thanks for the support, much appreciated. We're still far away from a playable state, realistically looking at the end of 2019, but we'll get there eventually :)

[SyrusDrake]

Hey, sorry for kinda hijacking this thread. I tried using the Python script kindly provided by you guys to decrypt the csv files on Linux and I either get the Error message

ValueError: Infile is not encoded with xtRLE

when trying it without specifying an output file or

ValueError: invalid literal for int() with base 10: ''

when specifying an output file.

Am I missing something?

[WizzardMaker]

OK, seems like I need to make the latest decryptor standalone.

Syrus, the python script was a direct port of a C script someone made for another game. We since have a much improved C# version which handles edge cases a lot better. I will push the standalone decryptor this/next week. You would need mono though to execute the program, as it's programmed in C#.

[SyrusDrake]

Cool, not sure if I'm smart enough to handle that but I'll give it a try as soon as it's out, thanks a lot ^^'

[WizzardMaker]

So, for anyone finding this, I can't post the decrypted CSV files on GitHub (for now, this can be discussed when implementing the routes etc.), but maybe you will find answers in the GoG forum for the game. cough

[WizzardMaker]

´*.raw´ files are the audio files of the game. They are standard 44100 KHz wav files. Example code:

audioFile = new AudioStreamSample()
byte[] data = System.IO.File.ReadAllBytes(PathToSound);
audioFile.SetData(data);
audioFile.MixRate = 44100;

A special sound class will be added to the repo soon.

[WizzardMaker]

Smacker files can now be read!

Thanks to a modified scsharp code. I will push my additions later, after polish.

Question, should we save these animations as files, so that we can drag them in, or should we use scripts and load them at runtime?

[Serphentas]

´*.raw´ files are the audio files of the game. They are standard 44100 KHz wav files.

Have you been able to play them on VLC or any other media software ? Even on hex editor, it looks far from a WAV (at least with my files).

[WizzardMaker]

Playback right now is working inside my forked smacker branch, which will soon be pushed to the main branch. To play those files outside of the game use Audacity and import the files as raw/unformatted audio files with 8Bit PCM and 44100Hz when using the /video/ files. The /SOUND/ files use 16Bit PCM and 22050Hz