Foul spawn may be causing horrible, inexplainable stuck bugs

[elmokki]

Generating 25 foul spawn nation batches has seemingly gotten stuck around 20 nations twice now. Running the last seed alone generates it reasonably fast. I got to go to bed soon so I recommend testing without pulling my branch in case the mage generation has caused this. I'm running one batch as last thing today.

There's a chance that instead of generating an array of nations we should generate one nation at at a time and write images and code between generating nations. That should free memory.

[elmokki]

Yup. Master branch got stuck too on 15th or 16th nation and the seed it got stuck on generated just fine. Please verify this isn't just me.

It's weird since people have been making mods of thousands of nations so it shouldn't be a memory issue. It pretty much has to be something related to memory. The alternative is that there is something that doesn't stay unmodified between nation batches and that eventually causes the program to get stuck.

One interesting thing would be to check if it gets stuck in the same place every time.

[elmokki]

Definition of stuck here is not managing to generate within a few minutes. That should be enough considering the nations generate relatively fast when not in a batch.

[elmokki]

...and lastly, if this won't get fixed, foul spawn need to be disabled for 0.6.11

[Ealb]

...I might have seen something like this before when working on FS, but if I did it was when I was having some inexplicable crashes as well so I didn't think too much of it. I'll do a sanity check for my standard size-40 stress test size though, Although hmm, 40 may be too many units if all of them are FS.

[Ealb]

Okay, I've been generating a nationset for ~30-40m now, and it seems pretty definitely stuck. It wasn't fast up to 20 or so, but it was still making progress. It was really slow to the point where I thought it was stuck by 27. It's been at 28 for at least 15m now, and maybe 20m. It's using 1.3GB of memory.

...and now it finished that and popped out two more in the blink of an eye. It's now on 31, and making no apparent progress again. I think I'll kill this and try to have a look at the nation it hung up on..

[Ealb]

Generating 40 nations with seed 1986440234. Generating nations Nation 0: 2016-06-12 19:24:58.098 Nation 1: 2016-06-12 19:25:12.259 Nation 2: 2016-06-12 19:25:27.917 Nation 3: 2016-06-12 19:25:39.48 Nation 4: 2016-06-12 19:25:53.872 Nation 5: 2016-06-12 19:26:04.404 Nation 6: 2016-06-12 19:26:10.264 Nation 7: 2016-06-12 19:26:25.552 Nation 8: 2016-06-12 19:26:43.741 Nation 9: 2016-06-12 19:27:00.586 Nation 10: 2016-06-12 19:27:23.078 Nation 11: 2016-06-12 19:27:36.092 Nation 12: 2016-06-12 19:27:54.77 Nation 13: 2016-06-12 19:28:25.649 Nation 14: 2016-06-12 19:28:43.703 Nation 15: 2016-06-12 19:29:01.904 Nation 16: 2016-06-12 19:32:06.74 Nation 17: 2016-06-12 19:32:29.952 Nation 18: 2016-06-12 19:32:45.829 Nation 19: 2016-06-12 19:37:26.876 Nation 20: 2016-06-12 19:56:16.953 Nation 21: 2016-06-12 19:56:23.019

I gave up at 20:39. There's definitely some distinct inconsistency here, though.

[Ealb]

Generating 21 nations with seed 1986440234. Generating nations Nation 0: 2016-06-12 20:45:50.41 Nation 1: 2016-06-12 20:46:06.041 Nation 2: 2016-06-12 20:46:24.027 Nation 3: 2016-06-12 20:46:36.99 Nation 4: 2016-06-12 20:46:52.913 Nation 5: 2016-06-12 20:47:04.679 Nation 6: 2016-06-12 20:47:10.774 Nation 7: 2016-06-12 20:47:27.687 Nation 8: 2016-06-12 20:47:47.408 Nation 9: 2016-06-12 20:48:05.894 Nation 10: 2016-06-12 20:48:30.739 Nation 11: 2016-06-12 20:48:45.08 Nation 12: 2016-06-12 20:49:05.113 Nation 13: 2016-06-12 20:49:38.99 Nation 14: 2016-06-12 20:49:58.875 Nation 15: 2016-06-12 20:50:18.736 Nation 16: 2016-06-12 20:53:37.878 Nation 17: 2016-06-12 20:54:01.724 Nation 18: 2016-06-12 20:54:18.397 Nation 19: 2016-06-12 20:59:13.468 Nation 20: 2016-06-12 21:19:25.692 Done!

Times are fairly consistent. Hmm.

[Ealb]

...and yep, that last one that was taking 20m takes less than 20s by itself.

[Ealb]

The way Nation 20 took about 20m, then Nation 21 took 7s makes me really think it's something like garbage collection freeing up resources, although it's weird that it immediately goes back to "even worse" for the one after that.

Still, I used to generate 300 nations at a go with no problem. Even if we assume FS nations are 5x as big (which probably isn't unreasonable), that still shouldn't result in grinding to a standstill after only 20 nations unless it's something like the data structures for FS as a race being huge and repeatedly duplicated, but even looking at the structures in memory in debug mode there doesn't seem to be enough to account for that kind of size.

[Ealb]

Hmm. Manually increasing the starting/max heap size slightly delayed the onset of long generation times, but only by a couple of nations, then the same wait times kicked in. It really does look like a memory issue even if it doesn't seem like it should be one given past batch sizes. I'll maybe try finishing Illithids and see if the same crap shows up with their hybrids.

[Ealb]

Hmm. 40 nations of montag-men took 5 minutes... although with them, naming took 5 minutes too. This really seems to be pointing to either the size or structure of Foulspawn data files.

[Ealb]

Following some testing, I now strongly suspect that the horrible and anti-NG way I initially designed size-2 infantry, ranged, and cavalry Foulspawn heads/helmets is to blame. I re-wrote FS cavalry heads to pick helmets first then #needstype a head based on the helmet selected (i.e., what the ogre-sized FS do) rather than all the convoluted and un-NG-yl #addthemeinc thisslottag (and/or #forceslot) stuff that I first did with little FS. Then I set all FS montag poses to pick nothing but cavalry and ogres (i.e., FS poses that do the cleaner selections of headgear). I generated 40 nations in 8 minutes, plus another 2 minutes for names, writing the .dm, previews, etc. I did a second test to make sure it wasn't a fluke and got another 40 nations in 8m + 2m. The last sanity check was trying 1986440234 from the above tests (I know seeds would be broken as far as specifics are concerned, but it'd still be closer than the random seeds). Again, 40 nations in 8m + 2m. It's possible that this is an effect of the nations just having a handful fewer poses, but dropping down from arbitrarily long/hanging nation generation to a fairly consistent ~12s average reeeeeeeeeally makes it look like the messed-up head stuff is to blame.

I'll re-write the rest of the heads tomorrow. Is it possible for me to give a single item multiple #needstype entries for a single slot? I.e., can I give the single item "no_helmet" "#needstype head canid", "#needstype head barehead", and "#needstype head hybrid"... or do I need to define "no_helmet_canid" "no_helmet_barehead" and "no_helmet_hybrid" and give each one a single #needstype? There's a very large amount of overlap with the small FS helmets, so the files will be fairly redundant (and data structures will be a lot bigger, though if I'm right about where our generation problem is coming from that's not a huge deal) if I need to make multiple versions of every helm. I tried to figure out this from a quick look at the code but it wasn't immediately obvious and I'm tired enough that I don't trust my judgement right now...

Also, is the average gcost/rcost feature for montag category poses working? It looks like the test nations I generated had gcost/rcost based on the empty stub structures when I commented out the fixed adjustmentcommand gcost/rcost,

[elmokki]

Is it possible for me to give a single item multiple #needstype entries for a single slot?

Multiple #needs and #needstype in single item should work fine.

Also, is the average gcost/rcost feature for montag category poses working?

There was a bug at some point that I fixed yesterday where the command to disable it was needed to enable it

[elmokki]

Oh wait, no, what you described for #needstype would just generate 3 helmets. I am not entirely certain on what you want, but from what I gather you'd want a random head from those three categories. If that is true, just add #needstype head whatever and #whatever to all the heads and it'd do the same (I assume themeincs were going to force the correct head?)

[elmokki]

Okay, I tried my primary solution, ie making the program write each nation directly after generating and not storing the nation. This did not fix the problem, which is kind of nice as it would need more rewriting to get descriptions etc working and would exclude backstories spanning multiple nations later on. However, I now truly hope that what you are doing will help.

It was also an unlikely solution considering how filling memory would be totally ludicrous.

[elmokki]

There was a roster generation bug that caused a lot of null-entries in an array in troopgen and is likely to have been problematic. While this technically wasn't foul spawn specific, it's very likely it in practice pretty much was considering how limited FS pose amounts are.

I wonder whether setting modules in nation null would help, though by all logic it wouldn't since they're defined only within generate() and should thus be handled by garbage collection afaik.

[elmokki]

On a side note, I think sacred poses should generate less than 10 options just for the sake of performance. 6 might be fine.

[elmokki]

I got to 25 and now am at 33. The speed does definitely slow down as 33 -> 34 seems to be progressing slowly. It has done all the montag stuff at like 7 minutes and still has stuff that is normally quite fast to do.

[elmokki]

Okay. If I disable montags from troops, all goes perfectly and very fast. Any performance issues are definitely caused by the troops now.

[elmokki]

...though it's 3 seconds or so on normal nations in early generation so it's hardly a consistent problem of course.

[elmokki]

Okay, I found one more, surprisingly glaring, performance issue. It still won't remove the problem, but 38 nations out of 40 generated quite fast, most of them generated the troops in ~3s

It is worth noting that 40 foul spawn nations is quite a bit more than 40 standard nations in terms of units, but still, this is a bit baffling. It is possible that the way foul spawn are constructed right now are problematic for the garbage collector, but primarily the code should be fixed since the system for generating them isn't THAT complex.

I suppose #addthemeinc would be possible to streamline to one filter that is referenced to in all units which may be a performance boost. That said, I've been tinkering with this for like 6 hours now (which is a good way to pass time though) and I have some other stuff to do. I'll try to implement that small potential performance boost later since it's a thing that could reduce object count by up to a few hundred or so, even if the objects are quite small.

There are a few other things too I bet. One is how races on nations became copies at one point and how that could technically be prevented by class NationRace which contains a reference to race and the special commands. As many objects, but the objects themselves contain less stuff. Overall all the getCopy() functions probably needs to be looked at, except for units and names where they at least have a good purpose.

[Ealb]

There are probably only 4-8 combinations of heads that can share a single helmet, so I'll just add more types (e.g. #type small_canid). The head data structures will be slightly larger, but it should still avoid the performance pitfalls (that I can't quite articulate - I feel like I should be able to describe why what I originally did with heads is so utterly awful, but I can't) with my head-determines-helmet implementation, and it won;t be anywhere as hideously redundant as making multiple copies of a given helmet for every head type that can use it.

[Ealb]

First test:

Generating 40 nations with seed -1100119381. Generating nations349827625 on 10:25:49 .735590767 on 10:26:11 .-761938258 on 10:26:24 .8404169 on 10:26:43 .1970178476 on 10:26:55 .-1196163466 on 10:27:06 .820552894 on 10:27:15 .602643837 on 10:27:29 .-1506262586 on 10:27:40 .-1292690249 on 10:27:59 .852413481 on 10:28:09 .1797685207 on 10:28:18 .1012729187 on 10:28:28 .2094546500 on 10:28:43 .1647984162 on 10:28:55 .-321994915 on 10:29:13 .876747402 on 10:29:31 4.-151838689 on 10:29:42 .-677345446 on 10:30:03 .1752467521 on 10:30:11 .-367481097 on 10:30:25 .-1967823548 on 10:30:37 .1889413303 on 10:30:54 .1600326666 on 10:31:05 .-1194452506 on 10:31:41 .961746412 on 10:32:01 .580116295 on 10:32:15 .2100124787 on 10:32:27 .-1931614415 on 10:32:43 .1670729488 on 10:33:46 .-1631848590 on 10:33:49 .1769181330 on 10:34:44 .-909049955 on 10:35:08 .-995498453 on 10:39:37 .-1704648849 on 10:46:28 .-51210180 on 10:46:46 3.-1307147665 on 10:46:52 .1768097092 on 10:47:06 .-2031915783 on 10:48:19 .2016633255 on 10:48:31

Ran smoothly until 33, at which point some were slow and some were fast. Overall much better though.

[Ealb]

Okay, so, further testing is still getting me some stalling behavior (particularly on the example I was testing last night - that looks like it was just an awful seed for whatever reason), so this wasn't the whole problem. It seemed to help, though. I'll go ahead and get your performance changes and see how those combine with this.

[elmokki]

I think the #addthemeinc tweak I will try to get done before I'll go to bed may also help substantially, but we'll see.

[elmokki]

Also if all else fails, we can technically do a super hacky solution where NationGen is restarted every 20 nations or so. Not that I really want to do that.

[elmokki]

It's not perfect but I'd guess your changes and mine together should help quite a bit. I mean, it was stalling around 20 yesterday and I can reach to reasonably fast 40 now.

If foul spawn basechance is like 0.1, they shouldn't really be causing any issues in even remotely feasible nation amounts. However, of course it'd be nice to find the cause of this. That said, the cause may be quite deep and something that genuinely needs rewriting a lot of stuff.

[Ealb]

Eclipse is driving me crazy. It claims to be unable to resolve MageMontagTamplate to a type even though it's being imported - although before it claimed it wasn't being imported until I cut-and-pasted the class name into the import line, at which point it suddenly recognized the identical import statement). It's probably something really dumb on my end, but I cannot figure out what it's doing.

Having said that, yes, unless we have someone generating massive amounts of FS nations the 0.1 should avoid most problems.

[Ealb]

Wow, yeah, that was Eclipse being very wonky. Copy-pasting the for-loops using MageMontagTemplate directly below themselves and deleting the "erroneous" for-loops fixed the errors.

[Ealb]

All right, looks good. My "problem seed" ran in 6m +2 m:

Generating 40 nations with seed 1986440234. Generating nations-685896431 (1) on 13:16:50 .-786570643 (2) on 13:17 .-132665159 (3) on 13:17:10 .-1801667656 (4) on 13:17:20 .783783524 (5) on 13:17:34 .400164253 (6) on 13:17:42 .-2068259934 (7) on 13:17:54 .-1741047853 (8) on 13:18:02 .2012060409 (9) on 13:18:13 .-968346803 (10) on 13:18:24 .461173847 (11) on 13:18:32 .-1080496068 (12) on 13:18:41 .-1343963167 (13) on 13:18:45 .-1804346018 (14) on 13:18:52 .-1371280255 (15) on 13:19:02 .-652838434 (16) on 13:19:12 .-1299632647 (17) on 13:19:18 .2008434771 (18) on 13:19:23 .-88036805 (19) on 13:19:29 .1848373412 (20) on 13:19:36 .-1687036850 (21) on 13:19:43 .1686310875 (22) on 13:19:51 .1578791705 (23) on 13:19:59 .83927082 (24) on 13:20:08 .-758932291 (25) on 13:20:17 .-1482694482 (26) on 13:20:26 .168526700 (27) on 13:20:32 .-1526266954 (28) on 13:20:39 .-1843015974 (29) on 13:20:49 .-46302308 (30) on 13:20:58 .-2118429245 (31) on 13:21:09 .-2116169367 (32) on 13:21:21 .1117052179 (33) on 13:21:31 .892687842 (34) on 13:21:35 .-1955111245 (35) on 13:21:42 Nation -1955111245 had a null filter for a mage. Nation -1955111245 had a null filter for a mage. .-262536015 (36) on 13:21:50 2.301787156 (37) on 13:21:57 .-1810863425 (38) on 13:22:18 .760561883 (39) on 13:22:28 .503487489 (40) on 13:22:35

I assume the null filter isn't good though.

Anyway, I'll upload my current branch. The basechances on the helmets will probably need adjusted since head distribution is no longer determined by head basechances though.