Displacement Bugs & Hammer Compile Error: CUtlLinkedList overflow

[MagnumMacKivler]

So this is a bit of a weird one. I've got a map project that has a lot of displacements, and when the map is compiled, some of the displacements are broken. I suspect that VBSP is running out of allocated memory for displacements.

Details

My map has exactly 6127 displacements as reported by the Hammer compile tools. compile_log_6100disps.txt

During the VBSP stage, I get a few errors, though these don't seem to affect anything noticeably: DRASTIC MEMORY OVERFLOW: Fell out of small block heap! make_triangles:calc_triangle_representation: Cannot convert (preceded by a ton of coordinates) I get these errors in just about every mapping project nowadays, they don't seem to cause any actual problems in the final map but I'm including them for completeness.

VVIS appears to go off without a hitch.

The real strange error comes when we get to the VRAD stage: CUtlLinkedList overflow! (exhausted memory allocator) (repeated thousands of times)

I've done a quick internet search for that error and it has been spotted in Source games before, but all the examples I could find were encountered in-engine, not in the compile tools.

The map successfully compiles and loads, but some strange issues appear on some displacements once in-game.

First, some of the displacements simply don't show up at all. Others only appear when looking at them directly. Now you see me... Now you don't.

These displacements also seem to ignore player physics, but only on parts of the triangle mesh. Vphysics seems to work okay, but players will fall through certain parts: As shown here, this is all the same displacement, and you can see the missing displacements in the background.

I think the problem is originating in VBSP because the player physics interactions are broken, even though the new error is from the VRAD stage. I think the VRAD errors are trying to reference data that doesn't exist because VBSP didn't put it into the BSP.

Workaround

I hid a bunch of displacements elsewhere in the map using visgroups, bringing the final total down to 4343. That got rid of all the CUtLinkedList Overflow errors and the same displacements shown above work perfectly fine, both in rendering and player/object physics. compile_log_4300disps.txt

My map has no leaks, all displacements are either Power 2 or Power 3. I am using the stock Gmod Hammer and compile tools. My Gmod version is x86-64 Branch, 2024.03.21.

I don't think it's an issue with hitting the displacement limit... That should be 16384 right? I know it has to be more than 4096 because the map with 4300+ displacements works fine. Also note that the working compile log has the Drastic Memory Overflow and MakeTriangles Cannot Convert errors as well, hence why I don't think they have anything to do with this.

My suspicion is that VBSP is running out of allocated memory for displacements, specifically, at some point during the compile process. My PC has 32GB of RAM available, but task manager indicates that Hammer isn't using more than 1.7 GB during the VBSP compile stage.

[robotboy655]

Map file? If you don't want to share it publicly, you can email it to me directly (address is in my profile), or make a test map file that reproduces the issue.

[MagnumMacKivler]

Sure thing! To save you the trouble of having to mount a metric ton of custom content, here's a sanitized version of my map with all the custom content removed. You'll need CS:S mounted though.

mags problem map files.zip

I've included the BSP and VMF so you can try compiling it yourself, if needed. I've been using the "Default" hammer advanced Run Map commands with the addition of -allowdynamicpropsasstatic for VBSP.

The same area shown above still has the issue even with all my custom stuff removed:

You can see it in person by teleporting here: lua_run Entity(1):SetPos(Vector(-10756, -11514, 8929))

And here's the compile log for the sanitized version, for completeness. compile_log_sanitized.txt

Thanks!

[robotboy655]

Ok, so the issue has nothing to do with VRAD at all, not running it entirely still produces the issues. The errors you get with VRAD appear to be related to internal "leaf" system.

It would seem that the root issue related to visleaves. Even without running VVIS, the missing physics seems to be only in certain visleaves, not at displacement boundaries. You can see this yourself with mat_leafvis 1.

I am not yet sure why this is, or how to fix it.

[MagnumMacKivler]

Interesting... It makes sense that it has nothing to do with VRAD. VBSP is the program that is responsible for making the visleaves initially, right? Perhaps the issue has to do with the map's BSP tree or it's making "corrupt" visleaves or something.

I will try poking around with the visleaves in the area to see if I can get it to behave differently. Perhaps if I can alter the space by adding world brushes, hint brushes, or visclusters, the issue may be worked around.

It is still odd though that removing geometry in other parts of the map fixes this problem area. If you compile with all the "tunnel" visgroups turned off, the area shown above works again.

[MagnumMacKivler]

So I tried a bunch of things to try and mess with the visleaves in the bugged area, like adjusting the nodraw floor height, adding world brushes floating in the air, and adding visclusters, but none seemed to fix the issue. Even with the visleaves completely recalculated with many different parameters, the visleaves in the area were still bugged.

I tried turning the displacements into regular brushes (func_detail in this case) and that fixes the solidity and rendering problems for the specific brushes involved. Neighboring displacements that were affected by the issue are still affected. Models also seem to work okay, so I have some more options for workarounds.

However, I did notice something else of note when checking other parts of the map... the bug seems to only occur in the northeast and southwest corners of the hammer grid, i.e. +X +Y and -X -Y. The center area does not exhibit the bug and the northwest/southeast corners don't seem to have them either. I haven't found any correlation with height, since the map spans most of the hammer grid from top to bottom and the problems seem randomly distributed along the Z axis. A clue perhaps?

[Battlefieldkille]

For the record; I tried to brute force the issue on a different map by adding 8k displacements with 30k+ numportals to see if there is any connection. I ran through the map with host_timescale 30 and ideally if there was a broken displacement - the player would fall through it. But everything worked fine, the issue that OP has is really bizarre. dispbruteforce.zip