Unused textures in HTS packs potentially not cleared from memory

[fzurita]

I'm getting complaints from a user about this specific texture pack in Android: https://drive.google.com/file/d/1nTLPa2Pyg1mX88xwEK754dBz1_hGDq1a/view

Apparently, this texture pack for Majora's mask is 18GB, there is no Android device or many PCs out there with that much memory.

Eventually, after playing for long enough, the app crashes, and I'm guessing that it's because of memory limitations.

I know I could look this up myself before making the issue, but this acts more like a reminder for me. Are textures from HTS files being removed from memory after they are no longer present in the cache? If they are not, that would likely lead to a crash once memory runs out.

[Rosalie241]

HTS doesn't really keep textures in RAM (at most there will be 1 texture stored in memory at the same time, see https://github.com/gonetz/GLideN64/blob/master/src/GLideNHQ/TxCache.cpp#L657), the issue you're most likely seeing is from the insanely high limit in Textures.cpp (where it loads textures in VRAM/OpenGL, the default limit is 8000 textures, see https://github.com/gonetz/GLideN64/blob/master/src/Textures.h#L109).

You could try https://github.com/gonetz/GLideN64/pull/2572 & https://github.com/gonetz/GLideN64/pull/2580 and tell users try to set the limit to 500-1000 or so.

[fzurita]

Interesting, yeah, I could see how VRAM limitations can cause the issue as well.

[gonetz]

insanely high limit the default limit is 8000 textures

8000 textures had not looked insane number until Nerrel's texture pack appeared. His textures are huge. Since there are no safe way to detect texture memory size with OpenGL, proper video memory management is problematic (impossible?).

[Rosalie241]

Since there are no safe way to detect texture memory size with OpenGL, proper video memory management is problematic (impossible?).

Maybe I can revise my #2572 PR to track MB's instead, i.e instead of knowing the exact usage, just track the upload data sizes of textures which we can get using TxUtil::sizeoftx and allow users to change that.

It wouldn't be perfect but at least it'd allow people to set some limit on how many texture data is stored in VRAM, even though it wouldn't be exact, it'd be better than no option at all imo.

[gonetz]

t'd be better than no option at all imo.

I agree, new option is less evil than crash. If we knew at least total VRAM size, we could use texture memory = VRAM/2 heuristic.

[gonetz]

BTW, I just found that @Nerrel finally released his Zelda MM texture pack. Nerrel, congratulations, incredible work! Thanks to @Rosalie241 too! It is pity though that the current version of the plugin is not recommended for the pack.

[Rosalie241]

It is pity though that the current version of the plugin is not recommended for the pack.

If you want, I can open a PR to restore the old behavior with an option which'll be disabled by default (something called Legacy texture mapping behavior), then Nerrel can recommend upstream GLideN64 again.

[gonetz]

May be later. I want the current texture mapping be polished first. Probably, the legacy behavior can be simulated with an uniform option in shaders.

[fzurita]

@Rosalie241 This pull request may act more like legacy: https://github.com/gonetz/GLideN64/pull/2593 can you check if it has the issues with the texture pack when using the "fast" path? The config parameter is currently only enabled for mupen64plus.

[fzurita]

@gonetz I think a fix for this is very much needed, at least for Android, where we have a very limited amount of RAM. I can't find a way for OpenGL to know how much total VRAM is present. But it can be found outside of opengl, for example in linux:

https://www.cyberciti.biz/faq/howto-find-linux-vga-video-card-ram/

In android, all I can find is that the GPU driver can take as much system memory it needs up to the full remaining space. So I believe it's just the RAM size minus some amount.

I would suggest we could do what @Rosalie241 suggests and she can update #2572 to be in bytes. Then the max bytes amount can be a configurable parameter. It will then be up to the front end to decide what the maximum amount is and provide that in configuration. I think this works well, at least for mupen64plus. For project64, the API would have to be updated, or the plugin itself would have to use some windows functions to find VRAM size.

[fzurita]

Actually, I think it would be fine without a config parameter. It could be calculated on the fly, you would just need a different method for each OS

Android: System RAM/2, system RAM can be retrieved like this: https://stackoverflow.com/questions/7374246/how-to-get-total-ram-size-of-a-device

Linux: Use one of these methods: https://www.cyberciti.biz/faq/howto-find-linux-vga-video-card-ram/

Windows: Use WMI: https://docs.microsoft.com/en-us/windows/win32/cimwin32prov/win32-videocontroller AdapterRAM attribute

[Rosalie241]

I personally think calculating it on the fly without a config option would be bad because we have no idea how much vram/ram is currently in use, so if you're using a different vram/rsm heavy application, it could crash again.

[fzurita]

Just doing some research. It appears that Android may just be a special case. In windows and Linux, once VRAM is fully utilized, the driver will start using system RAM, if system RAM is fully utilized, then the page file will go into use. So we may have to do nothing for windows and Linux where a lot of storage is available.

For Android, those things do eventually run out, so it makes sense to have a limit there.

Well, this may not necessarily be Android specific, but specific to devices with a low amount of resources.

[gonetz]

I think a fix for this is very much needed, at least for Android, where we have a very limited amount of RAM.

Ok.

Actually, I think it would be fine without a config parameter. It could be calculated on the fly, you would just need a different method for each OS

It would be great if we could automatically find safe-to-use amount of VRAM. As I can see, it is not that easy. It is straightforward only for Windows - call a WIN API method, parse the result. Android needs some work with activities, which can't be done on plugin's level. Linux needs to run a command and parse the result. It also does not look doable on the plugin's side. Correct me if I'm wrong.

I also found this article: https://stackoverflow.com/questions/4552372/determining-available-video-memory However, if it works, it works only on desktop (and not on Intel GPU)

There is another hint from https://stackoverflow.com/questions/3154632/how-to-measure-vram-consumption-on-android "Poor man solution: You could try to infer the VRAM size in an empiric way adding 1MB texture until you get an out of memory error from gl.glGetError()."

Can it be a solution: upload texture, check glGetError() - if we get an out of memory error then reset texture cache and lower texture cache size limit?

I personally think calculating it on the fly without a config option would be bad because we have no idea how much vram/ram is currently in use, so if you're using a different vram/rsm heavy application, it could crash again.

I don't see, how any texture cache limit can save us from this situation : "you're using a different vram/rsm heavy application". As I know, Android solves this situation by automatically closing inactive applications to free resources.

[gonetz]

Well, this may not necessarily be Android specific, but specific to devices with a low amount of resources.

Ok, we can add that option, but hide it from users on desktops. I'd like to check this strategy on mobiles first: "upload texture, check glGetError() - if we get an out of memory error then reset texture cache and lower texture cache size limit" @fzurita is it possible to check? Because I can't reproduce it on desktop.

[fzurita]

Well, in Android, what probably is happening is not that. With the texture pack, the app eventually takes too much RAM, so the OS kills the whole application.

Android will kill an application once certain RAM utilization limits are reached. I think this will happen before we get any GL errors.

[Rosalie241]

@gonetz

Can it be a solution: upload texture, check glGetError() - if we get an out of memory error then reset texture cache and lower texture cache size limit?

I don't think that's a good idea because on desktop (windows, linux), drivers map to system RAM when you're out of VRAM which'll be much slower to load/upload textures to as opposed to VRAM, and remember project64 is 32bit so I think that might crash due to the 4gb address space then.

[Rosalie241]

I think it'd be a better idea to just have sane standard config option values for the memory limit, i.e 1/2 gb for desktop, 1gb for android, etc, instead of trying to do some form of automatic detection because those solutions seem hacky in my opinion.

[fzurita]

To be fair, I don't think GPU driver memory utilization counts towards the 4GB limit of project 64.

[Rosalie241]

To be fair, I don't think GPU driver memory utilization counts towards the 4GB limit of project 64.

it doesn't, however when we're out of VRAM, it will map to RAM.

[gonetz]

I think it'd be a better idea to just have sane standard config option values for the memory limit

Ok, let's try to find that sane limit. As I understand, the problem is with Nerrel's huge hd textures pack. Let's find how many memory is necessary to load all textures for Termina fields for example. Or to all places inside the Town. It would be bare minimum for comfortable playing.

[gonetz]

I run Clock Town at night with Nerrel's pack. I did not visited all the shops (many are closed at night). Texture cache is 1600mb. It is only for textures, without frame/depth buffers, shaders etc. Windows Task Manager says that 2.7 GB of video memory is used (some takes desktop and other tasks). With fixed texture cache limit less than 2gb we will get constant textures reload, which along with file textures storage may result in poor performance. Poor performance is better than crash, but fixed cache limit of 1-2gb for graphics cards with 12gb of memory is too harsh. So, my suggestion:

• use current implementation with user-defined limit, without auto detection of RAM size, at least on desktop.
• 0 means no memory size limits, that is use current 'number of textures' limit.

[fzurita]

That sounds reasonable to me.

[weinerschnitzel]

RiceVideo had a feature to track texture usage to prioritize keeping some textures in cache. If impact of texture cache limit is noticeable with low end devices that would otherwise crash, juice may be worth the squeeze to do something similar

[gonetz]

RiceVideo had a feature to track texture usage to prioritize keeping some textures in cache.

GLideN64 puts new texture on the top of the texture cache. When the cache is full, the oldest texture in the cache is removed. So, often used textures are always on top. The problem is that even most recently used textures in Nerrel's pack take >1gb of video memory.

[gonetz]

@fzurita I finally checked the PR from Rosalie241 (yep, I'm slow nowadays). Could you check this branch with Nerrel's pack: https://github.com/gonetz/GLideN64/tree/Rosalie241-txMaximumAmountInVram ? I tested it locally with 1GB limit for hires textures. This limit is enough for one-two locations. Then it becomes exceeded, and the cache updates (removes some older textures) in every new location. It should not crash though.

[fzurita]

Yep, I'm on it. I'm also having a user that was experiencing crashes with it test it as well.

[fzurita]

This appears to be a success, the user that had the crashes no longer has them anymore. So in my opinion, it works great!

[gonetz]

Good to know, thanks! The fix committed to master.