Texture out of memory error (possible int64 wraparound)

[calpaca2]

Your system information

• System information: main components AMD Ryzen 7 4700u CPU, AMD Renoir Vega 6 GPU, 16GB RAM, OS is Ubuntu 20.04 with recent 5.12 kernel for better CPU support.
• Have you checked for system updates?: Yes
• Are you using the latest stable video driver available for your system? Yes
• Have you verified the game files?: Yes

Please describe your issue in as much detail as possible:

After about 20 minutes in-game the graphics significantly glitch because the textures do not fit anymore in the GPU memory (see this linked console dump from a bot game used as example).

The culprit seems to be an underflow in non-evictable memory. For example

[RenderSystem] TEXTURESTREAMING: Extremely low memory: Available mem: 8.33 MB, Required: 29.27 MB.
[RenderSystem] rendersystem (Vulkan) tracked resources:
[RenderSystem]   VB: 2311 buffers, 45.12 MB
[RenderSystem]   IB: 2289 buffers, 12.05 MB
[RenderSystem]   CB: 4226 buffers, 4.35 MB
[RenderSystem]   TEX: 5923 textures, 266.78 MB
[RenderSystem]   TOTAL: 328.29 MB
[RenderSystem] Texture Streaming:
[RenderSystem]   266 / 248 MB (non-evictable: 17,592,186,044,370 MB)
[RenderSystem] Overall System GPU memory use:
[RenderSystem]   2,984 / 3,307 MB

here we can see that non-evictable is 17,592,186,044,370 MB which is just shy of INT_MAX for an int64 (if julia's BigInt implementation did the math correctly). This is much more than my 3GB of GTT shared GPU memory, and textures are not loaded. Note that when this happens I usually still have 4-5GB of free memory (on top of the used GTT measured with radeontop).

Visually the ground and trees become untextured, and all units become "balooney" as the textures are not rendered. I will try to take a screenshot next time but for reference it looks something like this.

Restarting DOTA 2 solves the issue, until the next time the memory gets filled and represents itself. Leaving and joining games (to HUD) will not solve the issue. Once it is started, even re-entering another game or spectating tends to quickly trigger it again (I imagine because the overflow sticks around).

For video I am using a second monitor (HDMI connection) with main laptop monitor off and custom video setting (mostly taken from fastest preset) that can be seen

Steps for reproducing this issue:

1. Start a game of DOTA 2 (bot or multiplayer)
2. Play normally until graphical glitches start to appear

[Turmfalke2]

That would be an overflow, underflows only happen with floating point operations.

[calpaca2]

This is a pretty colloquially accepted expression, and I hoped from the context it would be pretty clear what I meant is happening. It seems the definition of over/underflow is much more ambiguous than I expected (e.g. integers cannot overflow in C11's standard). Still to be more precise I changed the title to fit wikipedia's definition and called this an integer wraparound, thanks.

[calpaca2]

As a short follow-up, I tried to see if the console command r_textures_evict_all would work as a temporary work-around. It does help because flushing all texture caches loads fresh textures in for objects around me, which fixes the visual glitches, but after a variable (2-10) number of minutes the textures glitch out again. Even right after r_textures_evict_all the non-evictable: 17,592,186,044,XXX MB wraparound is not reset (I do not know if the book-keeping is done on DOTA's side and should be affected or by the driver).

Thanks again for looking into this, and if any more debug info is needed let me know.

[calpaca2]

I am still experiencing this problem (same Extremely low memory and wrapped non-evictable memory), and I can also add that the wraparound can also trigger simply by idling on the main menu (I imagine since the Dawnbreaker dashboard is rendered using in-game engine). Moreover after an update (yesterday was not happening, today it was) the texture bug slightly changed (or maybe it is a new one, let me know if I should open a separate ticket).

After some in-game time (but before the wraparound) textures of some specific units (dire lane ranged creep, Harpy Camp, Large Satyr Camp and Hellbear Camp that I can remember of), and instead I can only see a red pulsating wireframe. I do not know if these two things are related (they are both texture errors). No console error pops up when only the red wireframe bug is present (I tried to use condump to get the whole log but cannot find it on disk). After a while, the wraparound kicks in and all textures are dumped, but the red wireframe remains.

To explain better, I could also finally could grab some screenshots when both bugs are active, and I am attaching them to the post. Hope they can help, if I can provide any other information let me know and thanks for looking into this.

Screenshot at 2021-05-16 01-01-56 Screenshot at 2021-05-16 01-02-27 Screenshot at 2021-05-16 01-16-10 Screenshot at 2021-05-16 01-16-17

[Turmfalke2]

(I tried to use condump to get the whole log but cannot find it on disk)

condump files should be created in dota 2 beta/game/dota/, but at least for me that isn't working.