TripleA slows down with large games/1 turn takes 20 mins for AI

[hispublic]

Engine version 1.9

My Operating System win 10 home 64..core i7 6700 32GB RAM 1080 graphics

Map name Happens on any large (many units) games

Can you describe how to trigger the error? (eg: what sequence of actions will recreate it?) play any map with large amount of units (I think)

Do you have the exact error text? no error text

Instead of this error, what should have happened?

Any additional information that may help

The attached saved game is from the latest version of Star Trek Dilithium War. This problem happens on any large game though. The engine just bogs down to a crawl. Sometimes it's so slow that the application goes into (Not Responding) for short delays. In the attached game, running on my somewhat powerful machine with the power setting to High Performance, waiting for the AI takes about 20 minutes per turn. Obviously this is not the intended pace of the game.

I made two configuration changes that have helped keep the delay to 20 minutes instead of hours...these changes are in the TripleA VMOPTIONS file: -Xmx10G -Xms10G

The system is currently using about 16GB of RAM, which means I should have plenty to play with, I think.

Does anyone have suggestions for making large games playable? This map in particular has "resource" locations that add/subtract bonuses from up to 100 units, so the author clearly intended this to be a massive game.

Thanks for any help.

[RoiEXLab]

@hispublic I don't see any "attached savegame", probably because the term is somewhat misleading, you need to link to it in your post. Also what exact version are you using? 1.9 is a too broad term. The exact version is shown when starting the engine in the upper left corner

[hispublic]

version 1.9.0.0.8304 saved game runs slow.zip

[hispublic]

@RoiEXLab I thought I had the save on the original...it should be on my second post. Please let me know if there's anything else you need...BTW, game turns are now up to about 40 minutes

[ron-murhammer]

@hispublic Can you try the latest pre-release version located here as there have been some recent AI performance improvements: https://github.com/triplea-game/triplea/releases. I'll try to take a look at your save game later tonight myself as well.

[ron-murhammer]

Oh wow that is a round 145 game... I've never seen anything past round 30 on any map LOL. Honestly the problem is the game data is so large because of the game history that the AI takes a long time dealing with it. I'd definitely recommend playing against Fast AI instead of Hard AI at this point. At the moment the AI and arguably the game engine really isn't designed to handle large maps that last 100+ rounds.

With the Hard AI each of the nations turns took about 3-5 minutes for me so a whole round would take 20-25 minutes.

[hispublic]

@ron-murhammer patience seems the only way to win as the Borg vs hard AI (game runs windowed while I do other stuff). I didn't know that each turn builds on the previous so that it's a race against turns...do I win first or does the PC crap out?..lol. So, is it the number of turns, the amount of units in the game, the type of AI or a mixture of the three? I was trying to make the game as hard as possible, so I used hard AI.

[ron-murhammer]

@hispublic Mix of all 3 but I think mostly just the sheer number of rounds. I think you could start a new round 1 game and edit in the exact same ownership/units/etc then it would probably run 10x faster. Notice your save game is like 8MB where most normal save games around <1MB.

Hard AI plays better than Fast AI but at a cost of taking more time to compute its turns. Depends on the specific map on how much better and how much longer.

[hispublic]

@ron-murhammer thanks for the explanation. I updated the engine and tried it...but it was still slow enough that I stopped it. After, I changed it to Fast AI and that made a HUGE difference.

I'm curious and if it's difficult to explain don't worry about it but does the previous turn info make any difference to the game, other than slowing things down? You mentioned that editing in the same units into turn 1 would work better, so why does the game keep the info in the first place? It seems that dumping this info can be a wish list item for a future release, if it's possible to do, as I've found this issue on a bunch of maps, as I tend to try to put myself into the most difficult situations and sometimes find myself in long games, especially on large maps.

[ron-murhammer]

@hispublic Well, the history let's you view past turns of both your own and your enemies which tends to be pretty important as you often in PvP games won't watch your opponents moves. Also I hope you have AI pause duration set to like 10 in the settings as otherwise just moving all the units could take a very long time. Generally the history isn't an issue as most games last less than 10 rounds and almost all games last less than 30. Honestly a 154 turn game is a huge outlier so I don't think the size of history was ever really consider for that.

[RoiEXLab]

@ron-murhammer Even if the round count is more than unusual, the AI shouldn't really take much longer than during the initial rounds. I'm not 100% familiar with the AI, but I'd assume the root cause lies somewhere in the OddsCalculator code. Where we modify a GameData object and undo the change.

[hispublic]

@ron-murhammer gotcha...that makes sense. Thank you again for your time and explanations.

With that explanation in mind, my revised wish list item would be a setting allowing the user to set the number of turns in the history, from zero, some to all turns. In theory, this would allow better single player game performance and bigger and/or more complex maps that might take many turns to complete.

Many of my games are longer than 30 turns, as I tend to play large maps and non-standard maps (greyhawk, waw PvWorld, etc) in single player, AI always set to Hard. This game is one of the longest, as I don't think the author intended a single player to choose Borg against the galaxy..they were meant as the spoiler player that ultimately is fodder...previous game as Federation ended quickly.

My AI pause is set to 0. I was trying anything to make it faster.

[hispublic]

@RoiEXLab If the OddsCalculator is attached to the Battle Calculator then I think you are on to something. In the saved game, the Battle Calculator is completely overwhelmed. At some point, it became futile (I'm playing the Borg, see what I did there) to try using the Battle Calculator because while it will show a count of units quickly, waiting for it to "copy data" is like watching paint dry.

[RoiEXLab]

@hispublic I just tested your savegame to see how well the performance is on my machine. Doesn't take that long for me (just up tp 15 mins), but still farr too long. Haven't identified the root cause yet, but the oddscalculator only calculates about 2 times per second which is far too slow

[RoiEXLab]

@hispublic That's a good hint: The battle calculator internally uses the same code as the Ai. If the cloning process takes really long, that could be another reason for it to "idle" for so long.

[DanVanAtta]

Should be the cloning process, we see the watching paint dry problem on WaW around round 15+.

Doing a more shallow copy of the game data, trimming off any data that we don't have to copy will surely help, as the problem is related to the amount of data copied. Overall the battle calc really does not need the full game data at all, just unit stats and territory effect, and also there really should not be a network copy from host to client when doing a battle calc either (the game data should just come from local).

These have been wish list items for a long time, improving the situation by reducing the amount copied even if we don't hit the ideal of sending just unit stats should be a big help.

[hispublic]

Thank You everyone for being awesome!! I just had to say it.

I'm curious, with the configuration changes that I've made, would it be the general consensus that I've made the best performance changes? Here's what I've done:

TripleA VMOPTIONS file (I have 32GB RAM): -Xmx10G -Xms10G

AI Pause duration set to 0

[RoiEXLab]

@hispublic Assigning more RAM to TripleA is ok, but you shouldn't assign uneccessarily much RAM to TripleA because this won't magically make TripleA perform better. The AI Pause duration shouldn't make a difference either. AFAIK this option is just to slow down the AI (like when using the Fast AI), so you can still follow it's moves and everything isn't too quick for you.

Technical Stuff

@DanVanAtta @ron-murhammer Ideally the OddsCalculator would use some sort of "inbetween object" that behaves like a game data object for battles, but really just is a cheap dummy object that only performs minimal synchronization to maximize performance and shares the read only information with other instances. I already tried making all GameData related objects implement the Cloneable interface in #2981, but I stopped when it got a little bit too big and the performance wasn't really better. I believe a problem is that every GameData related object holds some sort of backreference to it's associated GameData so easy cloning is simply not possible. Most of those objects don't even need this objects for more than one or two tasks, so ideally we'd restructure the code so we need to pass such an object to the according method to avoid the backreference. The biggest challenge is probably to come up with a good solution for this, a solution that avoids a lot of complexity and performs better performance whise.

[ron-murhammer]

@RoiEXLab I'm pretty sure really its just the game history that is 90% of the problem of long data copy times. Essentially, we'd just need to find a way to clear the game history before copying and then set it back. Then ensure any game history calls made by the battlecalc steps are essentially no-ops.

[RoiEXLab]

@ron-murhammer The problem is: How do we clear it before copying without clearing the actual history

[ron-murhammer]

@RoiEXLab Could we do the following:

1. Set history object to temp variable
2. Set data's history to a new empty history object
3. Copy data
4. Set temp variable back into original game data

[RoiEXLab]

@ron-murhammer I don't know, probably but this would of course come with the price of having to aquire a write lock instead of a read lock.

[ron-murhammer]

True but if having a write lock for 10% of the time that we had to have a read lock then its probably worth it.

[Cernelius]

@hispublic You can open task manager and take a look how much RAM your TripleA / java process is using. As long as it is using less RAM than your settings (in this case, 10 GB), then that is not the limit. I think it would never happen the RAM going above like 4 GB, but I've never played a 145 rounder.

[hispublic]

More data if anyone is interested in how the game is acting. I'm not asking anyone to look at it. I'm uploading if there is a glutton for punishment that would enjoy looking at the error log...or if it would help you with the tech discussion you were having.

At round 165 I got an error in the Battle Calculator regarding being out of memory, but got busy with something else and the console got hidden behind something (out of sight, out of mind) and I guess the errors have been piling up, even though I've stopped using the Calculator. When I saw the Console, the text was very long, so I zipped it, if anyone is interested in seeing it. The game seems to be running notwithstanding. Victory is imminent. long game Console errors.zip

[FrostionAAA]

@hispublic Woaa! :-o Round 165!!! That’s crazy :-D

I made this Star Trek map. I don’t think that it has ever been tested that intensely before, and I am pleased to see that someone plays this map.

Concerning the Borg, as this is a player with other units and stats than the other players, it would be great to hear your assessment about whether this player is properly balanced or if it needs unit stat changes, like a boost. LOL... I certainly did not intent it to be a 165 turn play as the Borg, but hey! As you you say, it is a chalange!

If you have feedback, it would be great to hear. I have made a thread on the forum for the map: https://forums.triplea-game.org/topic/840/star-trek-dilithium-war-official-thread

[ron-murhammer]

Gonna close this as I think we've made some improvement and more feedback on the map is probably better in the thread that @FrostionAAA created.