Turn 0 OOS - Pangaea Huge 18 AI

[Mouse]

Latest SVN version on both clients, Direct IP Connection, Cache cleared on both clients before startup each time, simultaneous turns disabled, random events disabled,

I am Client

Hosts perspective

We live in the same house playing on 2 different PC's. Here are the specs of both if it matters. Using a Netgear Nighthawk C7800 cable router/modem combo to connect. DxDiag-Host.txt DxDiag-Client.txt

no savegame available considering its turn 0. No OOS log, or I don't know where to find them.

[Toffer90]

Does it happen on World mapscript too? Tested any other mapscripts in this regard?

[Mouse]

We've only tried to play on Pangaea so far. What is the World mapscript? We're just casual players.

[MattOttawa]

Can you turn on python error messages in bts config file and repost the first screenshot?

[Toffer90]

No OOS log, or I don't know where to find them.

should be here: USERNAME\Documents\My Games\Beyond the Sword\Logs

[Mouse]

loggingenabled = 1 hidepythonexceptions = 0

I hope those were the correct settings. Here are each of our logs directories Logs-Client.zip Logs-Host.zip

[Toffer90]

These two as well perhaps: synchlog = 1 randlog = 1

Edit: I can see that you are getting different starting location plots. But something may have gone wrong before that point. Would be nice to know if this is related to the mapscript or not.

Another thing is that the NPC teams (animals, barbs, neanderthals) are not initialized as alive players for one of you.

[Mouse]

loggingenabled = 1 hidepythonexceptions = 0 synchlog = 1 randlog = 1

Here are the new log files. Logs-Client.zip Logs-Host.zip

[Toffer90]

This time it is clear that you got different maps, which makes the pangea mapscript a big suspect.

Strange that there's so many areas on a pangea map though, must be lakes aplenty I guess.

[Mouse]

Is there any other information I can provide to help or is it all up to the code now?

[Toffer90]

Is there any other information I can provide to help or is it all up to the code now?

Could you check if it happens on the world mapscript too?

Another thing, have one of you deleted the arboria mapscript? Just me being curious, probably ain't the cause in any case.

[Mouse]

We're both on revision 11343. Just updated to make sure. No changes.

[Mouse]

We tried a match on C2C_World as well. Same issue.

Map settings:

Here are the logs Logs-host.zip Logs-Client.zip

[Toffer90]

Map generation is obviously playing out differently for you two. OOS issues are quite complex to debug. The info you provided might be helpful.

For now you two just have to accept that when starting a new game you both have to restart the game and load the initial save from turn 0 that the host created after map gen.

Interesting to see that you have identical river generation, so landform must be the same, Bonus placement is the first point the logs reveal that stuff is out of sync, so maybe that narrows it down adequatly.

[Mouse]

This time, after running on C2C_World, both the client and the host got spammed with Python Exception message boxes. The PythonErr.log file is quite large on both devices now. The host had to restart the computer because the process hung. We don't know if the host's log is complete. The client's log should be complete. Same settings as before.

Logs here: Logs-Client.zip Logs-Host.zip

Image taken from host computer before the process hung:

[Toffer90]

both the client and the host got spammed with Python Exception message boxes.

Yeah, there was an error in the OOS logging code, I fixed it on git now, will be in next SVN. The OOS logger code isn't the cause of the OOS though, it is only run after OOS has happened.

Could you guys test something for me, both of you add this test mapscript into your privateMaps folder, and try starting a new MP game with that script; and upload the logs again? C2C_World_OOS_TEST.zip

[Mouse]

The Python exceptions did not occur that time. But it still went OOS on turn 0. Here are the settings we used for that (default):

Here are the logs and the savegames as well just in case they can help. Logs-Host.zip Logs-Client.zip OOS_Savefiles_CLIENTHOST.zip

Yeah, there was an error in the OOS logging code, I fixed it on git now, will be in next SVN.

Edit: Also, that's cool. I'm glad we could at least help with that!

[Toffer90]

You don't have to screenshot the settings anymore btw. Saves probably won't be helpful in this case.

Could you do the same test again with this test script? (yes I added different debugging to it this time) C2C_World_OOS_TEST.zip

[Mouse]

Alright. And here you go: Logs-CLIENT.zip Logs-HOST.zip

[Toffer90]

This is just weird, you guys are out of sync even before map generation has started...

[Mouse]

What could that mean? Should we reinstall our copies of Civ and C2C and try again just to make sure, or is it definitely a code issue?

[Toffer90]

What could that mean? Should we reinstall our copies of Civ and C2C and try again just to make sure?

Try doing the SVN→Cleanup command, right click SVN folder to find it. Check all boxes in the pop up for it. Have you guys shared the UserSettings folder, so that is the same for you both?

Could you test one last time first though, added debug lines to the absolute first code line in the mapscript that is processed when launching the MP game. C2C_World_OOS_TEST.zip

[Mouse]

Yep. Here are the logs for that. We're going to do the cleanup command now and we'll post logs again afterward: Logs-Client.zip Logs-HOST.zip

[Toffer90]

Heh, at the very beginning you are in sync; good that I found a point where that was the case. May I call on you two to do more test another day, it's getting real late for me here (01:35), and the next test would have to involve me compiling a new dll for you two to use.

[Mouse]

Of course. And here's the results of the last test after the cleanup. Also, The cleanup command removed the UserSettings folder for both of us and it was regenerated by the game. We'll check back tomorrow. Thanks for taking the time.

Logs-CLIENT.zip Logs-HOST.zip

[Toffer90]

Thank for helping out.

The cleanup command removed the UserSettings folder for both of us

They should then be identical, so no need to share them at that point.

[Mouse]

Okay. Just for consistency, we shared the usersettings folder between computers. Host to client and still got an OOS on turn 0 have a good night :) Logs-CLIENT.zip Logs-HOST.zip

And a final test run with a clean install, including a removal of all folders, files, configs, and profiles related to BTS and C2C, and a new checkout of the SVN. It would seem after comparing logs between host and client myself, that it could be due to unit placement. There are X and Y values between logs that don't match, as well as gold calculation for Lincoln, Menelik II and Pachacuti. The logs also diverge at the end at line 244238. The client has no more entries after that line. I'm no expert on the engine, but that just seems strange. I'm just trying to learn to read the logs for myself. Logs-HOST.zip Logs-CLIENT.zip

[Toffer90]

Ready for new test? Here you go: TEST.zip

Edit: Deleted some redundant comments to make the issue page a bit more tidy.

[Mouse]

Using the DLL, we can no longer connect to each other and the host's game hangs when he tries to exit the lobby after I attempt to connect and timeout. The default game still functions as expected, so it's not networking settings on our end that we know. The network log on the host does have one error line:

"We are broadcasting a public host, but have not initialized public host tools!"

This is the host's network log: network-2056-HOST.txt

[Toffer90]

That's strange, all I did was add some extra logging. Well, I also compiled a debug dll instead of a regular release compile, meaning a lot of extra "debug" code that is normally ignored by the compiler is included in the dll. Here's a release dll with the extra logging: CvGameCoreDLL.zip Edit: Oh and btw; update to SVN rev 11344 before using this dll.

Our debug dll must have worked in MP at some point as it has specific multiplayer debug code as well. @Thunderbrd did you ever try debugging OOS/MP games with a debug dll compile back in the days? E.g. this multiplayer specific log file is only written when using debug dll:

[Mouse]

Alright, we can connect again. Updated like you said using the new DLL.

Getting a python error before the map finishes loading: C2C_World_OOS_TEST:4331 ('GC' is not defined). Logs included Logs-CLIENT.zip Logs-HOST.zip

[Toffer90]

Again? • TEST.zip

Edit: Have both of you set your steam BtS version to the beta version? You should both do that if you haven't. In this case beta version means final release version of BtS. Steam made their own patch to the game that just mess things up.

[Mouse]

We've been using the release_original_unsupported like we're supposed to. The newest map and dll you sent loaded all the way until the OOS on turn 0. Here are the logs we have.

Logs-CLIENT.zip Logs-HOST.zip

[Toffer90]

Ok, I've pinpointed where you guys go out of sync; now I have to figure out why and how to fix it.

    for (int iI = MAX_PC_PLAYERS; iI < MAX_PLAYERS-1; iI++)
    {
        PlayerTypes ePlayer = (PlayerTypes)iI;

        FAssertMsg(!GET_PLAYER(ePlayer).isAlive(), "NPC is not expected to be alive at this point of the game initialization.");

        if (ePlayer == BEAST_PLAYER)
        {
            addPlayer(
                ePlayer, (LeaderHeadTypes)GC.getDefineINT("ANIMAL_LEADER"),
                (CivilizationTypes)GC.getDefineINT("BEAST_CIVILIZATION"), false
            );
            GET_PLAYER(ePlayer).setNewPlayerAlive(true);
            GET_TEAM(BEAST_TEAM).init(BEAST_TEAM);
            GC.getInitCore().setTeam(ePlayer, BEAST_TEAM);
            GC.getInitCore().setHandicap(ePlayer, (HandicapTypes)GC.getDefineINT("BARBARIAN_HANDICAP"));
        }
        else if (ePlayer == PREDATOR_PLAYER)
        {
            addPlayer(
                ePlayer, (LeaderHeadTypes)GC.getDefineINT("ANIMAL_LEADER"),
                (CivilizationTypes)GC.getDefineINT("PREDATOR_CIVILIZATION"), false
            );
            GET_PLAYER(ePlayer).setNewPlayerAlive(true);
            GET_TEAM(PREDATOR_TEAM).init(PREDATOR_TEAM);
            GC.getInitCore().setTeam(ePlayer, PREDATOR_TEAM);
            GC.getInitCore().setHandicap(ePlayer, (HandicapTypes)GC.getDefineINT("BARBARIAN_HANDICAP"));
        }
        else if ....

You are in sync before the for loop, and out of sync after it.

[Mouse]

Interesting - So, as animals and barbarians are being initialized and set alive? Are those counted as players? I just want to know.

[Toffer90]

Interesting - So, as animals and barbarians are being initialized and set alive? Are those counted as players? I just want to know.

Yeah, something is obviously not MP friendly with the way we initialize the NPC factions, and yes they count as players/teams in their own right.

[Toffer90]

Another test? CvGameCoreDLL.zip

[Mouse]

Here are the logs with the new DLL. Still OOS turn 0. Logs-Client.zip Logs-HOST.zip

[Toffer90]

Here are the logs with the new DLL. Still OOS turn 0.

Thanks, still working on pinpointing it more precisely.

[Toffer90]

Another round. • CvGameCoreDLL.zip

Getting closer.

[Mouse]

Happy to help. Here's the next round of logs.

Logs-Client.zip Logs-HOST.zip

[Toffer90]

Thanks; You guys go out of sync on this line: GET_PLAYER(eNewPlayer).initInGame(eNewPlayer, bSetAlive); For the first NPC initiated in the loop I mentioned

Of course that is another function with more lines to pinpoint within. I have a feeling this function wasn't meant to be called before game is properly initialized, due to the name "initInGame", this happens even before mapscript has started generating the map, so I wouldn't call that being in a game yet.

[Mouse]

That sounds like it could be a potentially critical bug.

[Toffer90]

Again: CvGameCoreDLL.zip

[Mouse]

There ya go. Logs-Client.zip Logs-HOST.zip

[Toffer90]

Again: • CvGameCoreDLL.zip

Still no attempt to fix, I think I'll have enough info after this test to do so.

[Mouse]

And here ya go! Logs-Client.zip Logs-HOST.zip

[Toffer90]

Does this fix the OOS? ♦ CvGameCoreDLL.zip

If not, upload logs as usual.

[Mouse]

C2C_World_OOS_Test - Turn 10 Normal speed C2C_Pangaea - Turn 10 Normal speed C2C_World - Turn 10 Ultrafast

We built multiple units per turn, had all of them moving separately and automatically, built multiple gatherers and had them automate city improvement separately as well.

No OOS. Looking good.

[Toffer90]

Nice.

Thanks for sticking with the program helping us pinpoint this bug. It was a quite recent OOS introduced when I figured out how to stop exe from allowing you to define more than 40 players in the staging room game setup screen. Slot 40-50 are reserved for NPC and stuff gets wonky if you in staging room assign a normal leader/civ to one of those slots.