Improve GTA4's pixelated shadows.

[RaphaelK12]

Hi, I'm looking to improve GTA4's pixelated shadows. This isn't exactly an issue here, but something I'd like to fix in GTA4 that I haven't found a solution anywhere and I need help from someone who knows about GTA4 and/or wrapper for d3d9. I analyzed all shaders looking for how to activate the linear filter for the shadow, but I didn't find anything about TEXTURE SAMPLE STATE, so I believe that shader effects were not used and SAMPLE STATE was assigned via code in the executable itself. I'm looking for some way to create a wraper to see all the SAMPLE STATE calls for the game's textures and see which one doesn't assign LINEAR as a filter and can replace it with linear. I'm not entirely sure that the problem with pixelated shadows is just SAMPLE STATE assigned as NEAREST or POINT, but from my experience with OpenGL I believe that's just the problem.

[RaphaelK12]

In the last week I've been doing a lot of research on hlsl, fx and d3d9, I found a good decompiler of GTA4 shaders (rage shader editor), so I was able to change the filter type from POINT to LINEAR (I was wrong, fx was used), but the result was not what I expected.

Click to expand (Images 1)

Shadow with original square pixels:   Filtro atualizado para LINEAR  Different sampling distance (less distance). 

Analyzing the shader assembly, I noticed that the shadow is compared in relation to the distance of the projection and a comparison is made that returns 0 or 1, that's why the shadows have cuts (which is softened with 12 samples from nearby locations and averaged).

Click to expand (Images 2)

Point 1 sample  Linear 1 sample  Point 9 sample  Linear: 9 samples  Expected result with 1 linear pcf sampling  Expected result with 9 linear pcf sampling 

To make this soft shadow in GTA4 we need linear sampling, use tex2Dproj to do the pcf sampling and have the shadow texture created like

pd3d9Device->CreateTexture(SHADOWMAP_SIZE, SHADOWMAP_SIZE, 1, D3DUSAGE_DEPTHSTENCIL, D3DFMT_D24S8, D3DPOOL_DEFAULT, &gShadowZSamplerDir, NULL);

but apparently it was created using: (D3DFMT_D16 , D3DFMT_D32, D3DFMT_R16F or D3DFMT_R32F) in place of D3DFMT_D24S8 and D3DUSAGE_RENDERTARGET in place of D3DUSAGE_DEPTHSTENCIL, which causes the sampling to treat the shadow map as a texture and not as a shadow map, thus returning only a depth value and not a comparison if it is in shadow or not.

I've been doing several tests on the shaders to see if it's possible to make soft shadows without changing the texture format, I've been getting some results close to what I expected, but unfortunately with the introduction of some glitches:

Click to expand (Images 3)

Tests:        Glitches:      

The glitch that has occurred in some places like the wooden fence, the shadow occurs before the fence, so it is darkened by the shadow itself (shadow acne has also occurred in several places), and other places the shadow of distant objects is supersaturated making them pixelated again (reverse effect of PCSS, where PCSS makes the shadows of distant objects blurrier and the shadows of nearby objects sharper).

To do better without increasing the amount of shadow samples (affects performance) the only way is to be able to modify the shadow texture creation to:

pd3d9Device->CreateTexture(SHADOWMAP_SIZE, SHADOWMAP_SIZE, 1, D3DUSAGE_DEPTHSTENCIL, D3DFMT_D24S8, D3DPOOL_DEFAULT, &gShadowZSamplerDir, NULL);

and make few modifications to the shader, being possible even to add smooth shadows equal to GTA5 (NV PCSS), only problem is that I don't know how to modify, I only know that the texture name is "gShadowZSamplerDir".

[xowny]

dude!!!! ur doing god's work, for a long time a thought that this wasn't possible but here we are, i hope someone with more knowledge join you ( i would if i knew code).

On Sat, Oct 8, 2022, 15:33 RaphaelK12 @.***> wrote:

In the last week I've been doing a lot of research on hlsl, fx and d3d9, I found a good decompiler of GTA4 shaders (rage shader editor), so I was able to change the filter type from POINT to LINEAR (I was wrong, fx was used), but the result was not what I expected. Shadow with original square pixels: [image: image] https://user-images.githubusercontent.com/38438130/194717475-ad432592-3537-420d-b2fc-0e77b35928ab.png [image: image] https://user-images.githubusercontent.com/38438130/194717551-786f6fab-5d59-4e5c-bca8-0aca2a78f1b5.png Filtro atualizado para LINEAR [image: image] https://user-images.githubusercontent.com/38438130/194717699-857678f8-987f-4ac9-a14a-052be8f9ce72.png Different sampling distance (less distance). [image: image] https://user-images.githubusercontent.com/38438130/194717591-b48a21da-2a1d-4314-8931-ac3718a497cd.png

Analyzing the shader assembly, I noticed that the shadow is compared in relation to the distance of the projection and a comparison is made that returns 0 or 1, that's why the shadows have cuts (which is softened with 12 samples from nearby locations and averaged).

Point 1 sample [image: image] https://user-images.githubusercontent.com/38438130/194719143-147e6086-9959-4c10-bc31-fceef4142238.png

Linear 1 sample [image: image] https://user-images.githubusercontent.com/38438130/194719182-59e41385-33bb-4959-a71a-130aded2fefe.png

Point 9 sample [image: image] https://user-images.githubusercontent.com/38438130/194719102-158537de-178d-4690-8485-24d3f472ec81.png

Linear: 9 samples [image: image] https://user-images.githubusercontent.com/38438130/194719113-0242de42-0552-4ba1-a7c1-a68b13f60dab.png

Expected result with 1 linear pcf sampling [image: image] https://user-images.githubusercontent.com/38438130/194719354-2f4a04a8-3516-4d8d-abe5-195c0c821ce2.png

Expected result with 9 linear pcf sampling [image: image] https://user-images.githubusercontent.com/38438130/194720033-0739386e-2574-463b-b7d8-8ec950aa57cb.png

To make this soft shadow in GTA4 we need linear sampling, use tex2Dproj to do the pcf sampling and have the shadow texture created like

pd3d9Device->CreateTexture(SHADOWMAP_SIZE, SHADOWMAP_SIZE, 1, D3DUSAGE_DEPTHSTENCIL, D3DFMT_D24S8, D3DPOOL_DEFAULT, &gShadowZSamplerDir, NULL);

but apparently it was created using: (D3DFMT_D16 , D3DFMT_D32, D3DFMT_R16F or D3DFMT_R32F) in place of D3DFMT_D24S8 and D3DUSAGE_RENDERTARGET in place of D3DUSAGE_DEPTHSTENCIL, which causes the sampling to treat the shadow map as a texture and not as a shadow map, thus returning only a depth value and not a comparison if it is in shadow or not.

I've been doing several tests on the shaders to see if it's possible to make soft shadows without changing the texture format, I've been getting some results close to what I expected, but unfortunately with the introduction of some glitches: Tests: [image: image] https://user-images.githubusercontent.com/38438130/194721117-5a479f53-91ae-433c-a8a5-17efc89a94c0.png [image: image] https://user-images.githubusercontent.com/38438130/194721136-737c8c32-0c52-4cd0-b7ce-66711e13522e.png [image: image] https://user-images.githubusercontent.com/38438130/194721150-f88dc990-dcd0-4e6e-a5a4-59ca85cac187.png [image: image] https://user-images.githubusercontent.com/38438130/194721162-116b8960-f9cc-48a9-9752-4d9822f55269.png [image: image] https://user-images.githubusercontent.com/38438130/194721207-56e221c1-5e88-4c5b-91e0-3106cd20d956.png [image: image] https://user-images.githubusercontent.com/38438130/194721230-70bce41e-5e7a-43ae-936a-80c6d24aaa4a.png [image: image] https://user-images.githubusercontent.com/38438130/194721242-c953784a-e4c7-49a1-ac65-d22f2d4b8d07.png

Glitches: [image: image] https://user-images.githubusercontent.com/38438130/194721295-147fa10e-a1b6-499b-be31-1b3306989d08.png [image: image] https://user-images.githubusercontent.com/38438130/194721310-e16a6a0e-cd3a-469c-b3ad-1f61f247a9b9.png [image: image] https://user-images.githubusercontent.com/38438130/194721322-d68a2308-36d8-4ed5-8548-9b3fa2616be6.png [image: image] https://user-images.githubusercontent.com/38438130/194721375-b775f65b-8229-441f-81ed-90e67e0f040a.png [image: image] https://user-images.githubusercontent.com/38438130/194721388-3e2c70e1-ff41-4d29-a47c-95ef7479780a.png [image: image] https://user-images.githubusercontent.com/38438130/194721396-595d930a-d964-4c4f-b8f4-3ccbe9817250.png

The glitch that has occurred in some places like the wooden fence, the shadow occurs before the fence, so it is darkened by the shadow itself (shadow acne has also occurred in several places), and other places the shadow of distant objects is supersaturated making them pixelated again (reverse effect of PCSS, where PCSS makes the shadows of distant objects blurrier and the shadows of nearby objects sharper).

To do better without increasing the amount of shadow samples (affects performance) the only way is to be able to modify the shadow texture creation to:

pd3d9Device->CreateTexture(SHADOWMAP_SIZE, SHADOWMAP_SIZE, 1, D3DUSAGE_DEPTHSTENCIL, D3DFMT_D24S8, D3DPOOL_DEFAULT, &gShadowZSamplerDir, NULL);

and make few modifications to the shader, being possible even to add smooth shadows equal to GTA5 (NV PCSS), only problem is that I don't know how to modify, I only know that the texture name is "gShadowZSamplerDir ".

— Reply to this email directly, view it on GitHub https://github.com/ThirteenAG/GTAIV.EFLC.FusionFix/issues/56#issuecomment-1272375053, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALW3M754ZMUZY5CWYQSO2JLWCG5ANANCNFSM6AAAAAAQ5EUAG4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[ThirteenAG]

If you need to replace specific CreateTexture, I can suggest using any d3d9 wrapper and setting a breakpoint at CreateTexture, detect the one you want and see where it's called from in the debugger.

There's also a tool called Helix Mod, which allows to "browse" through shaders via numpad, save their assembly, and load this assembly which can be modified.

[RaphaelK12]

Thanks a lot ThirteenAG, this dx wrapper is super easy to use, I thought it was more complicated, it's rare to download, compile and run without needing to edit something to make it work.

I spent today testing, I found 2 textures that are used for shadow, I modified it to use the correct values ​​for shadow, so far with one of them everything is in shadow, the other without any shadow, I'm "lost" at the moment and I don't know if it is a problem in the shader or if there is something between the moment of creating the texture until the moment of being used that I am missing, I have not found what is wrong. In the examples that come in the DXSDK_Jun10 package I also couldn't get the "shadow mapping" example to work with PCF (it's probably a problem with the matrix or texture in this case), only emulating with shader that reduces the performance very quickly from 1500fps to 90fps.

I don't have much knowledge of Directx, but I still believe it is something related to the texture format, one of them is created in D3DFMT_R32F format, another in FOURCC_INTZ format, so I researched this is a format used to read the depth buffer like a normal texture ( that's why in the shader it uses the distance comparison with the texture value). The light projection matrix in the viewing space seems to have problems as well.

I will keep researching and testing, at some point we will have decent shadows, with dx wrapper it was easy to discover a lot, I just need time and learn more.

[RaphaelK12]

I analyzed a lot on how GTA4 uses DirectX9 and how the shadow textures creation process was done, I found a lot of interesting things, but I discovered bad things about the way DirectX9 works and the way the game used it.

In low shadow quality the game creates 2 textures of 512x512 pixels and 1 of 1024x256. In medium quality, multiply the resolution by 2, high by 4 and very high by 8.

#define SHADOWMAP_SIZE 512      // example for low
IDirect3DTexture9* TexRender = NULL;        // Used only to create the shadow renderTarget
IDirect3DTexture9* TexDepth = NULL;     // depth texture, shadows rendered in
IDirect3DTexture9* TexCascadeShadow = NULL; // cascaded shadow map
pd3d9Device->CreateTexture(SHADOWMAP_SIZE, SHADOWMAP_SIZE, 1, D3DUSAGE_RENDERTARGET, D3DFORMAT(D3DFMT_R32F), D3DPOOL_DEFAULT, &TexRender, NULL);
pd3d9Device->CreateTexture(SHADOWMAP_SIZE, SHADOWMAP_SIZE, 1, D3DUSAGE_DEPTHSTENCIL, D3DFORMAT(MAKEFOURCC('I', 'N', 'T', 'Z')), D3DPOOL_DEFAULT, &TexDepth, NULL);
pd3d9Device->CreateTexture(SHADOWMAP_SIZE * 2, SHADOWMAP_SIZE / 2, 1, D3DUSAGE_RENDERTARGET, D3DFORMAT(D3DFMT_R16F), D3DPOOL_DEFAULT, &TexCascadeShadow, NULL);

The TexRender texture apparently has no use, the TexDepth shadow texture is created in a format not suitable for shadows (at least not for hardware PCF shadows, but necessary to be used in texture copy), the TexCascadeShadow texture I was ignoring at first until I understood what it was used for, first GTA4 renders the close shadow on the TexDepth texture, then copies the result to the first 1/4 of the TexCascadeShadow texture, then renders the second part of the shadow and copies it to the second quarter of the TexCascadeShadow texture and so on 2 more times. Something interesting is that the TexCascadeShadow texture is half the height and twice the width, so the TexDepth texture is copied and rescaled at half resolution to TexCascadeShadow, this rescaling works as a kind of anti-aliasing (although it is inappropriate and does not work well).

Example of TexCascadeShadow in front of the first house under the train tracks: From left to right: Near Shadow, Near Middle Shadow, Far Middle Shadow, Far Shadow.

I would like to use PCF shadows by hardware, but the TexCascadeShadow texture format is not favorable and the way the TexCascadeShadow texture is created is also unfavorable, because if it is changed from D3DFMT_R16F to some D3DFMT_D32, D3DFMT_D24X8 or D3DFMT_D16 format the copy of TexDepth does not work, DirectX9 has a lot of limitations regarding the handling of texture formats and does not allow hardware PCF filtering for formats that are not depth specific.

I would like to use PCF shadows in hardware because it uses less texture samples to smooth the shadow, no hardware pcf is needed 4x more samples reducing performance greatly.

At the moment I can only emulate in shader using 16 samples:

Click to expand (Images)

4 samples (like simple bilinear interpolation):  At the moment it only works for low quality shadows, but I intend to add to other levels or find out how to make it work regardless of quality (shadow texture resolution). Medium:  High:  veryhigh 

[RaphaelK12]

DONE. https://github.com/RaphaelK12/GTA4-Improved-Soft-Shadows

Thanks for lending me a good dx-wrapper.

New improved versions will come.