Help with FBS definition testing

[yesod30]

Hi, i wanted to help in writing the FSB files, so i decided to take a look at the two current one that are in the repository, and use the flatc compiler to use them to get the json from the game files. I extracted the .dat files with AssetStudio from the respective ab file (i tried with charword_tableaa5320 ), but when i try to execute the command for getting the original json file from the binary file, no json was generated. The command i used is .\flatc.exe --json .\charword_tableaa5320.fbs -- .\charword_tableaa5320.dat The fbs file was copied from this repo. Can you help me in understanding what i'm doing wrong to generate and check the if a FBS definition is correct?

[newOzer]

After some testing, it looks like the binary file has a 128 byte offset before the root table. I was able to verify the schema by generating the schema code and then passing an offset of 128 to GetRootAs().

Python example:

Generate python files with flatc --python charword_tableaa5320.fbs -o CharWordTable

from CharWordTable.Root_CharWordData import Root_CharWordData

buffer = bytearray(open('charword_tableaa5320.dat', 'rb').read())
root = Root_CharWordData.GetRootAs(buffer, 128)

And from there, you can verify the root object correctly parsed the binary with something like root.CharWords(0).Key(), which should return b'char_102_texas_CN_001' as per the first entry in charword_table.json
To convert the binary directly to a JSON using flatc, you would have to first remove the leading 128 bytes of the binary file before running the same command you are currently using. Alternatively, you could find a way to specify the 128 byte offset in the flatc call, but I don't see any mention of that in their documentation.

[StarHeartHunt]

After some testing, it looks like the binary file has a 128 byte offset before the root table. I was able to verify the schema by generating the schema code and then passing an offset of 128 to GetRootAs(). Python example:

Generate python files with flatc --python charword_tableaa5320.fbs -o CharWordTable

from CharWordTable.Root_CharWordData import Root_CharWordData

buffer = bytearray(open('charword_tableaa5320.dat', 'rb').read())
root = Root_CharWordData.GetRootAs(buffer, 128)

And from there, you can verify the root object correctly parsed the binary with something like root.CharWords(0).Key(), which should return b'char_102_texas_CN_001' as per the first entry in charword_table.json To convert the binary directly to a JSON using flatc, you would have to first remove the leading 128 bytes of the binary file before running the same command you are currently using. Alternatively, you could find a way to specify the 128 byte offset in the flatc call, but I don't see any mention of that in their documentation.

The first leading 128 bytes are RSA signature from Hypergryph to ensure the integrity of tables.

[newOzer]

The first leading 128 bytes are RSA signature from Hypergryph to ensure the integrity of tables.

Ok, that makes a lot of sense.

Is there an easy way to have flatc ignore these bytes? Or is the best option to just trim them before using flatc?

[StarHeartHunt]

The first leading 128 bytes are RSA signature from Hypergryph to ensure the integrity of tables.

Ok, that makes a lot of sense.

Is there an easy way to have flatc ignore these bytes? Or is the best option to just trim them before using flatc?

You can just trim that before sending it to flatc 👌

[yesod30]

The initial RSA signature seemed to indeed be the problem. Removing that now correctly create the json file. For that, i wrote a Powershell script to automate the whole process of removal and launch of flatc.

# Script accepts as first argument the ABSOLUTE path of the fsb, as secon d argument the ABSOLUTE path of the .bytes file exported by AssetStudio
# Requires the flatc exe to be in the same folder of this script
Try
{
    $file = $args[1]
    $bytes = [system.io.file]::ReadAllBytes($file);
    $filePath = Split-Path -Path $file;
    $fileName = (Get-ChildItem $file).BaseName;
    $newFileName = $filePath + "\" + $fileName + "_NoRSA" + ".bytes";
    [System.IO.File]::WriteAllBytes($newFileName,$bytes[128..($bytes.length)]);

    $flatc =  "$PSScriptRoot\flatc.exe --json --raw-binary " + $args[0] + " -- " + $newFileName + " --strict-json --defaults-json --natural-utf8 --no-warnings"
    Invoke-Expression "& $flatc"

    $jsonFile = $filePath + "\" + $fileName + "_NoRSA" + ".json";
    if ([System.IO.File]::Exists($jsonFile))
    {
        Write-Host "Json created"
    }
    else
    {
        Write-Host "Failed to create Json, fsb schema is probably not correct" -ForegroundColor Red
    }
}
Catch
{
    Write-Host "Failed to create Json" -ForegroundColor Red
    Write-Host $_.ScriptStackTrace -ForegroundColor Red
}

[yesod30]

For the third point of the verification, the one where it's said to check the official JSON JsonPropertyAttribute and JsonIgnoreAttribute, how are those extracted?

[yesod30]

Another addition: the conversion from Flatbuffer to json removes all fields with a default value. If the json will be used for loading the data for various application, the flatc command needs to add the --defaults-json flag, so those values are written too. This is useful also for testing, since Enums can be temporarily put as int, to then be mapped. I updated the previous script to include the flag

[StarHeartHunt]

For the third point of the verification, the one where it's said to check the official JSON JsonPropertyAttribute and JsonIgnoreAttribute, how are those extracted?

Those are extracted from the game's definition in global-metadata.dat by tools like Il2cppDumper. But global-metadata.dat and libil2cpp.so in Arknights APK are protected by MTP (libtprt.so) and it needs reverse engineering experience to restore the structures. For now, you can also just simply compare the JSON structure from flatbuffer with the previous one to verify.