Fix the Streaming Memory

[PlatinMTA]

Is your feature request related to a problem? Please describe. Right now it's impossible to run MTA with some HD texture packs without having some streaming problems. Even 256x256 texture packs that only change certain buildings can cause your game to lag when loading new textures or simply refuse to display them at all.

This is a known issue that is not solved by just setting the streaming memory to maximum. A lot of people choose alternatives to MTA simply because they can't have their custom HD mods. This also makes it impossible to replace every texture of San Andreas with an HD (or sometimes not even HD) texture packs by the server. The only alternative for doing that is using shaders, which could lead to low download speeds because of the size of certain textures. Binary reading is hard to do (altough not impossible) but using .txd directly should be the optimal solution.

Related: #2205 Forum Post dating back to 2010: https://forum.mtasa.com/topic/24022-a-stream-fix-feature-request/ (in the second page of that post is a link to the sourcecode of one of the streaming memory fixes, but it seems that this code is already used by MTA).

Click me to see what I'm talking about


Describe the solution you'd like A stream memory fix should be integrated in the options menu, one that lets even more memory be allocated for textures. There are like 25 asi's arround gtaforums that can do this trick, one of them was made by fastman in 2011.

Describe alternatives you've considered None.

Additional context I don't know the logistics of this and if its even possible, feasible or worth it. I know that we already have an option in advance settings to set the Streaming Memory, but it's not enough for some of todays players needs. Custom .asi's make this a non-issue, in comparison the MTA counterpart is kinda weak and does not fix the issue in its totality.

Someone with knowledge about this topic should check how much this can be improved. Clients that mod their game will highly appreciate a fix for the flickering, lag at load and disapearing buildings when using HD textures.

I couldn't find any issue that is directly targeted to this (maybe #2205?). I know there were some issues in the mantis waaay back but the problem was with MTA Streaming, not with the streaming in general.

[Dutchman101]

MTA is designed for original versions of GTA SA, and we have always reiterated that support for (heavily) modded installations isn't guaranteed (when it comes to stability, crashes and performance) and principally not provided.

I have seen some pretty intrusive and high quality mods work fine in MTA, so when the issues you have described are exposed the client must be using extremely high detail / total overhaul mod packs. Well, if it's easy to raise the limits fair enough, but it wouldn't be too bad to just fall back on the MTA vs GTA mods standpoint that i referred to.

[gta191977649]

Furthermore, using a shader replaced texture may result in conflict with other shader that applied in this texture. E.g. roadshine probably doesn't work after replaced the texture by the shader of road. However if you did through enginereplaced methods won't lead to this issue. So i would say definitely a good enhancement for the future MTA release.

[PlatinMTA]

MTA is designed for original versions of GTA SA, and we have always reiterated that support for (heavily) modded installations isn't guaranteed (when it comes to stability, crashes and performance) and principally not provided.

I have seen some pretty intrusive and high quality mods work fine in MTA, so when the issues you have described are exposed the client must be using extremely high detail / total overhaul mod packs. Well, if it's easy to raise the limits fair enough, but it wouldn't be too bad to just fall back on the MTA vs GTA mods standpoint that i referred to.

I do agree that MTA shouldn't provide full support for modifications made by the client to their own game, that would be silly. That being said I think this should be considered an exception if the task at hand isn't that hard to come by, because it's purely cosmetic and can be something that MTA server owners (just like the guy that made this issue #2205) could exploit for the benefit of everyone playing on their server.

My problem is that there isn't really that much room to have even some of the slightest type of texture overhauls on your game, which is a shame considering that a lot of players do use this types of modifications without any problem in SP or even in some MTA alternatives.

At least I don't think there are that many issues with low texture resolutions that only change some parts of San Andreas. The same can be said about high poly cars.

The point i'm trying to make is, If there is not a drawback by "fixing" this issue, then I will argue that it's totally worth it to add it and a big portion of the playerbase would be glad, including some server owners.

[ccw808]

Increasing the streaming memory means less memory for MTA/GTA which can cause out of memory crashes. AC will block some img textures being replaced if they increase the transparent content. e.g. Foliage

[Dutchman101]

Increasing the streaming memory means less memory for MTA/GTA which can cause out of memory crashes.

Maybe there is a way to detect if the user has a dedicated GPU or not? Because it will only decrease usable process memory if that memory is dynamic (due to it being integrated GPU, like most laptops and Intel HD on desktops)

AC will block some img textures being replaced if they increase the transparent content. e.g. Foliage

Even with increased streaming memory, we don't need to guarantee that all mods/total overhauls will work perfectly fine.. there would still principally be no official support.

[Pirulax]

In theory we just have to adjust the value of CStreaming::ms_memoryAvailable (0x8A5A80). By default (with the streaming memory slider in MTA on maximum) 256 MB. But, AFAIK this is only used for GTA streaming. Now with #1677 it would be nice to have a function to change the streaming memory limit, because with large models this PR doesnt work (models flicker, as GTA tries to free up memory).

[Dutchman101]

In theory we just have to adjust the value of CStreaming::ms_memoryAvailable (0x8A5A80). By default (with the streaming memory slider in MTA on maximum) 256 MB. But, AFAIK this is only used for GTA streaming. Now with #1677 it would be nice to have a function to change the streaming memory limit, because with large models this PR doesnt work (models flicker, as GTA tries to free up memory).

If we give power to the server for deciding a client's streaming memory, we may not need to care about those "running out of RAM faster" concerns as much. As long MTA doesn't force it upon an user, servers can make many mistakes that are on their end. Server owners should just be careful with such a feature, and may even estimate how high-end the user's PC is (with a function like dxGetStatus) before deciding to set a higher limit, or to decide which limit to set. And who knows the RAM savings that #1677 brings will even offset the impact on running out of memory with onboard GPU's (if server raises the streaming memory limit) more easily.

Long story short, i believe that a function for opt-in can be a short term solution, which means implementing it now and then keeping this issue ("Fix the Streaming Memory") open, awaiting some technique to safely raise streaming memory in general (for all players with a suitable PC, with a way for MTA to detect if it's a smart move - e.g my previous comment about detecting if they have a GPU with dedicated video memory or not, and otherwise if their RAM is sufficient to have room left after it increases the dynamic GPU memory). So that we'll still strive for a global solution.

[Pirulax]

MTA won't even run on a PC with 1 gig of ram for example. I think the minimum is around 1.5-2 gigs for freeroam. According to the source code, the maximum stream memory is 256 MB (thats for 2 gigs of RAM). If we add this function it could be raised sky-high (and the function is needed in order #1677 to function). Obviously, scripters should take precautions when using this function as too low limit will cause bugs, (but no crash should occur though), and too high limits might cause crashes (as the process might run out of RAM).

[Dutchman101]

MTA won't even run on a PC with 1 gig of ram for example. I think the minimum is around 1.5-2 gigs for freeroam. According to the source code, the maximum stream memory is 256 MB (thats for 2 gigs of RAM). If we add this function it could be raised sky-high (and the function is needed in order #1677 to function). Obviously, scripters should take precautions when using this function as too low limit will cause bugs, (but no crash should occur though), and too high limits might cause crashes (as the process might run out of RAM).

Yeah, well i meant that in my eyes we can just add the function first.. MTA scripters can already mess a lot of things up, the added responsibility of being careful enough with such a function is probably no different to this principle.

If we wanted to enforce proper practise within servers (or limit the amount of damage scripters can do by design), we're already too late anyways. We're probably fine to just add a big warning on the wiki page for that new function, telling about the risks and precautions needed.

I am just saying "add the function, no big deal" because i dont expect anyone wants to invest time in creating a good method to detect how far streaming memory can automatically be raised (per PC). Then, server owners that want to utilize your new "load custom IMG containers" feature, can avoid issues if they want and know how to handle it safely.

[Pirulax]

We could add available system memory to dx stats or something. It's pretty trivial.

[Dutchman101]

We could add available system memory to dx stats or something. It's pretty trivial.

Good idea.. bonus would be to also add a variable that indicates whether user has a dedicated GPU, or onboard (with shared memory). So that the careful scripter knows if he actually has to be careful to begin with (if it'll affect client RAM at all, to raise streaming memory limit)

I guess Windows has an API to differentiate that.

We can work towards some sort of AI using the same principles later (to apply higher but safe streaming memory limits for all users), but for now this will do. Even if we just provide available RAM to the scripter.

[Pirulax]

I think this returns the correct amount of memory (even with an iGPU) https://github.com/multitheftauto/mtasa-blue/blob/3f9348106feb1e64757fe92a06f6973bf3decd23/Shared/sdk/SharedUtil.SysInfo.hpp#L297-L320

[PlatinMTA]

Sadly #2264 didn't fix the issue at hand. Textures not loading and freezes (which make the loading wheel appear) are still ocurring, even when the Streaming setting is at maximum (which wasn't the case by default I might add). Examples: https://imgur.com/a/wQS14BF

I'm testing with the lowest resolution of the RoSa Project 256 (a complete remaster of San Andreas textures, some versions use AI while others supposedly do not). The modifications itself weight arround 600MB and are quite heavy, but even with the changes to the options there weren't any improvements.

Things to note:

• SA (gtasa.exe) doesn't seem to use more than 1GB of memory.
• I'm using a heavy gamemode to test this changes, which adds ped skins, object models and changes some zones. Even then I'm 99.9% sure this isn't why I hadn't seen any improvements between r20912 and before.
• The server was not updated to r20912, and it's using r20739.
• For some reason, after loading some textures, they will start failing to load and the game will display their LOD counterpart. If you go far enough, nothing will be rendered (as shown in the images on imgur: https://imgur.com/a/wQS14BF). Setting the player position through setElementPosition() seems to fix this issue.
• When loading new textures, the ram usage is the same. Disk usage, however, seems to go up. Maybe there is a bottleneck? Not sure because I'm pretty sure this textures work flawlessly with an streaming_fix .asi in singleplayer and SAMP.

[Dutchman101]

Sadly #2264 didn't fix the issue at hand

Good to know for that mod, but the benefits can vary per scenario. We don't know everything there is to know about that particular mod, or if something else in it causes problems.

[Pirulax]

What you linked is a delayed loading of objects. There was an attempt to fix this with CModelCacheManager, back in the day.

This is a GTA streaming issue, which hasn't yet been fixed, and probably would require the entire GTA streaming engine to be refitted (a fix may be to add more channels (currently there's only 2) to load data).

I'm not an expert in the streamer, but what I presume is happening is that the streamer just doesn't load stuff from the queue fast enough, even even though it could (as, for example, my game is on an SSD, yet GTA just doesn't utilize it)

Maybe you could try to use engineRestreamWorld although I'm not 100% sure if it's in the current code, or its just in a PR (img open pr most likely). If it's only in a PR, its very easy to add separately. Also, what you could try is my new function, as it clears the streaming memory (you can also see the streaming mem usage in /showmemstat)