How to extract data

[Istador]

Add a readme to the repository that explains how to extract data from the game.

I've already written a short readme, but the wording is terrible (for non developers), the tools are awful and it takes several hours or too much manual involvement.

---

For the moment, this might be helpful for you:

• They are located in .\Pine_Data\StreamingAssets\bundles\
• The .manifest files can be greped to find out what is in which file.
• The files are compressed .asset files - they may need to be decompressed depending on your tool.
  • UABE can only open one compressed file at a time, but offers batch decompression (but no batch export :/ ).
  • UABE needed ~5-10 minutes to decompress all but ~9-12 hours to open all .asset files at once (=> open only the ones you need, based on the .manifest).
• Game Objects and Transforms are separate from each others and reference to each other via Path-ID and File-ID.
• Be careful when dealing with the Path-IDs. These IDs are 64-bit signed integers. Not all tools handle them well.
  • jq gives you back wrong values without an error or warning.
  • UABE has the Path-ID signed in the extracted JSON and unsigned in the filename.
  • unsigned2signed: echo $unsigned | awk -M '$1>=2^63{$1-=2^64}1'

[Istador]

There is now a bash script in the export branch that guides through the manual extraction steps and supports it with automated actions to ease the process.

The directories are somewhat hard-coded and require a similar shell and dependencies installed (mainly: bash, cygpath, jq, unix2dos, dos2unix, awk, sed, grep, sort, uniq).

I run it inside the Git Bash from Git for Windows and only installed jq manually (needs to be in $PATH).

---

The things that I already extracted are also under version control inside the raw/beta_1 diectory

[buergi]

I extracted all food and material resources positions and numbers. I used uTinyRipper to batch extract all resources by calling uTinyRipper *.assets and hitting the extract button. Then I used a horribly badly written python script to extract the positions as well as the number of items.

Unfortunately one needs to know the number of resources one can harvest from a single source, which is almost always 2, the only exception I know of is Beagalite. Therefore the numbers might for some resources by off by a factor 2.

Here pine_resources.zip is the script as well as the resource dump.

I noticed that the IDs as extracted by UABE and uTinyRipper seem to differ. Therefore and since the files in raw/beta_1 do not contain the number of resources I did not make a pull request.

If I got time the next days I'll add the stuff to the google spreadsheet.

[buergi]

Just finished adding all resources to the spreadsheet.

[Istador]

Thanks for that, I never heard of uTinyRipper before. I only tried UABE and UnityEx.

When you add the data to the spreadsheet, please leave out the Animal stuff for now (Puffle Eggs, Bleeker Antenna, etc.).

I was thinking that it'd be better to only add the nests of the creatures as an own category than the individual items (that should spawn in the nests). But I wanted to check if both coordinate sets overlap first. [Edit:] Better add both, Nests and Items.

[Istador]

The number of items is encoded deeper inside the MaterialSource / FoodSource on the MonoBehaviour Base that is responsible for the item pickup. We should probably also export the ItemID.

[Istador]

With the new Beta 2 version, it's a lot easier to find and extract the coordinates of Collectibles and Resources.

Because they are all together inside of F8B1C847249A489BB4FA6F880F5600C1 instead of being separated over all bundles.

• Assets/Scenes/AlbamareScene/Areas/Resources/Resources_02_Mountains.unity
• Assets/Scenes/AlbamareScene/Areas/Resources/Resources_01_ForestPlains.unity
• Assets/Scenes/AlbamareScene/Areas/Resources/Resources_04_Dunes.unity
• Assets/Scenes/AlbamareScene/Areas/Resources/Resources_03_Swamps.unity
• Assets/Scenes/AlbamareScene/Areas/Resources/Collectables.unity

The only thing that is more difficult than it was before (currently I only checked the emblems) is that they now have a parent Game Object whose Transform position isn't (0, 0, 0). So, all coordinates need to be adjusted by that vector.

Export scripts need to be adjusted to check for parent transformations and applying them.

---

The export via uTinyRipper doesn't seem to contain the values of the MonoBehaviour properties. So item amount and id is not included in the extract :(

Maybe over the CLI it can be given an option to export more data, but I doubt it. When UABE exports a MonoBehaviour, the dump contains this data:

{
    "0 MonoBehaviour Base": 
    {
       [...]
        "0 ItemIngredient itemIngredient": 
        {
            "0 ItemID id": 
            {
                "0 int value": 67
            },
            "0 int amount": 1
        },
       [...]
    }
}

But now that all the things that we are interested in is in one file that only takes 10.7 MiB extracted instead of being separated over 3.94 GiB of data, it also seems feasible to just extract everything with UABE in one manual step and then go over it with a script that traverses the tree.

The Path-IDs from UABE's export and the ID's from uTinyRipper (which also seem to be Path-IDs) have totally changed. Probably because they are only unique inside of the same bundle or asset. They don't seem to be stable over different game versions, meaning we shouldn't export them but need to compare coordinates (guided by software, showing us similarities and differences).