Map Tearing

[mrosack]

See generated video here: https://www.playyourdamnturn.com/game/67b8acab-9162-4edf-a8f8-6f40dc8990f8... I'm seeing output like this pretty frequently and I'm trying to debug the root cause but it's slow going. If you notice frame 1 actually looks great, but it goes bad really quickly. Here's a couple saves from that game, 000002 is the good one and 000085 generates the tearing.

saves.zip

[rcannood]

Oh dear!

It appears there's been an updated to the civ6save binary format. This code is responsible for converting the binary data in the civ6save file into something that I can work with.

Guess we'll have to figure out which additional information is included in which use case.

[rcannood]

Could you also send me 000001?

[mrosack]

2 is the earliest i have - the user plays the first turn, uploads it, and that's saved as turn 2. Here's 3 if that's easier to diff.

000003.zip

[rcannood]

Thanks!

Unfortunately I'm not able to open 000085.Civ6Save locally because I don't have all of the Civs that you have. Would you be able to send me a screenshot of the minimap? I'm particularly interested in the region from where the tearing starts ;)

[rcannood]

I created a repository (rcannood/civ6save_analysis) with some information on how I started reverse engineering the tile information in the Civ6Save file. I created it mostly for myself (because I had to spend some time getting back into the format since it was already so long ago since I last looked at it), but it should also provide some helpful info for getting started with adding more information to the Civ6Save Data Sheet google document.

[mrosack]

Whoa, that's some great info, thanks! I should have some time to dig in this weekend, and that will really save me a lot of time getting into it. Here's a screenshot of 85:

[rcannood]

Aha! So it must be the aurora that's causing this :)

[rcannood]

Could you give me links to the mods you are using?

I tried the following, but they seemed to be incorrect.

• Better Balanced Game 5.3.2 (incorrect)

• Better Balanced Starts 2.2.0 (OK)

• Julius Caesar (Rome) (incorrect)

Edit: it seems I can unlock the Julius Caesar DLC by linking the game to a 2K account somehow. Guess I'll try that :)

[mrosack]

You're right about julius caesar, this is what i have for bbg: https://steamcommunity.com/workshop/filedetails/?id=2865001760

[mrosack]

I don't know if this helps, but something I've been playing around with is looking at this other project that seems to have a slightly different view of how things work: https://github.com/SamuelH91/Civ6EGRM/blob/master/saveFileHandler/filehandler.py

From that i tweaked the pattern to this which seems to grab on to the data in that file a bit better, but I'm running out of time to look at this today and I haven't tested anything in the javascript:

struct Tile {
    u32 travel_regions;
    u32 connected_regions;
    u32 landmass;
    u32 terrain;
    u32 feature;
    u16 natural_wonder_order;
    u32 continent;
    u8 number_of_units;
    u32 resource;
    u16 resource_boolean;
    u32 improvement;
    s8 improvement_owner;
    s16 road;
    s16 appeal;
    s8 river_e;
    s8 river_se;
    s8 river_sw;
    u8 river_count;
    u8 river_map;
    u8 cliff_map;
    Flags flags1;
    u8 ownershipBuffer;
    u8 unknown;
    u32 tileOverlayNum;

    if (tileOverlayNum == 1) {
        u8 buffer[24];
    } else if (tileOverlayNum == 2) {
        u8 buffer[44];
    } else if (tileOverlayNum == 3) {
        u8 buffer[64];
    }

    if (ownershipBuffer >= 64) {
        u8 oBuffer[17];
    }
    /*TileFlags2 flags2; // Ownershipbuffer
    Flags flags3;
    TileFlags4 flags4; // tileoverlaynum
    Flags flags5;
    Flags flags6;
    Flags flags7;

    if (flags4.co2) {
        u8 co2_byte0;
        u8 co2_byte1;
        u8 co2_byte2;
        u8 co2_byte3;
        u8 co2_byte4;
        CO2Flags co2_flags;
        if (co2_flags.mountain_improvement) {
            u8 mountain_improvement_byte0;
            u8 mountain_improvement_byte1;
            u8 mountain_improvement_byte2;
            u8 mountain_improvement_byte3;
            u8 mountain_improvement_byte4;
        }
    }
    if (flags2.owned_by_player) {
        u64 city;
        u32 district;
        u8 owner;
        u32 world_wonder;
    }*/
};

s128 start = 0x0010B933;
Tile tile[100] @ start;

[rcannood]

Apologies for the first version that I provided you with :sweat_smile: I was exhausted at the time and the proposed hexpattern was definitely not correct (as you noted as well).

There's an updated version at https://github.com/rcannood/civ6save_analysis/blob/main/patterns/civ6save.hexpat which fixes some of the issues. I'll check whether I can merge my version with yours.

Nice job finding Civ6EGRM! I... guess I can stop reverse engineering the civ6save format now? :p

[mrosack]

Great, thanks, I'll give it another shot tomorrow!

[mrosack]

I put the "overlayNum" stuff into the parse map helper.js and it seems to have solved the tearing, but I haven't tested it much. I'm also thinking we should maybe port over some of the feature edge cases they've got in there because it seems like they affect the buffer and would probably introduce more tearing, but if this handles 95% of tearing I'll be happy. I need to do some more testing with it this weekend, though.

https://github.com/pydt/civ6_pipeline/tree/bugfix/tearing

[mrosack]

OK, I think I've got a pretty good workaround in that branch (and my main now) based on an idea from the Civ6EGRM - testing the next tile to find a valid terrain to make sure we're in sync. I've got some test cases in the main PYDT API (https://github.com/pydt/api/tree/master/_docker/civ6_pipeline/node_handler/_testdata) - I'd like to get them in here but i was just struggling too much to figure out the best way to do it, so if you have ideas feel free and I'll put new tests in here going forward.

[rcannood]

I put the "overlayNum" stuff into the parse map helper.js and it seems to have solved the tearing, but I haven't tested it much. I'm also thinking we should maybe port over some of the feature edge cases they've got in there because it seems like they affect the buffer and would probably introduce more tearing, but if this handles 95% of tearing I'll be happy. I need to do some more testing with it this weekend, though.

Hmm.. This implementation fixes 000085.Civ6Save, but unfortunately it now causes tearing for the AutoSave_0162.Civ6Save that's already in the civ6_pipeline.

---

OK, I think I've got a pretty good workaround in that branch (and my main now) based on an idea from the Civ6EGRM - testing the next tile to find a valid terrain to make sure we're in sync. I've got some test cases in the main PYDT API (https://github.com/pydt/api/tree/master/_docker/civ6_pipeline/node_handler/_testdata) - I'd like to get them in here but i was just struggling too much to figure out the best way to do it, so if you have ideas feel free and I'll put new tests in here going forward.

That's a great idea actually! :+1: :D This could be a great workaround until we figure out the buffer issues, and even afterwards in anticipation of Firaxis potentially releasing additional content. Can you send a PR to this repo?

---

In the meantime, I've been busy trying to create a map with the worldbuilder that contains one of each terrain type, feature, continent, resource, world wonder, improvement (@Grifs idea): https://github.com/rcannood/civ6save_analysis/tree/main/testmap. Hopefully I can get as many different objects in one save file so we can use that to figure out the correct buffer lengths and test our code.

[rcannood]

Merged the branch I should've merged a while back!

Just as a heads up, grifs and i managed to make a lot of progress in deciphering parts of the civ6save to where we can reliably figure out the buffer for all of the save files we currently have available (and more). However, the parsing is now done using the imHex pattern language, so you may need to install additional dependencies in your custom container at some point.