Support for fixing NVIDIA texture transparency bug in DxWnd

[ghotik]

Hi. I'm sorry to raise a question about a problem that likely you fixed already, I hope you will support my request. I'm trying to fix a NVIDIA texture transparency bug that affects some dark and opaque textures in Rainbow Six (1998 edition) with Direct3D rendering. These textures (namely, some windows of a building) in black color become transparent. According to your git log it seems that the problem matches the description and fix here: https://github.com/narzoul/DDrawCompat/commit/32301efd5ce80ee5189b136ceb6b2dd4ec81b4e9 . The fix implies hooking the D3DDDI callback pfnSetTexture where you apparently just set some methods, probably to trigger some actions elsewhere. I'd be very grateful if you could spend a few words trying to comment the rationale behind your code to save me some time, blind importing of unnecessary code and a sure headache. If you like, you can find references of my discussion on DxWnd forum here: https://sourceforge.net/p/dxwnd/discussion/general/thread/eade78ae84/?page=3#5711

[narzoul]

It's a rather simple workaround. It checks all the texture stages, and if none of them have a texture set that uses color keys, then it force-disables color keying via D3DRENDERSTATE_COLORKEYENABLE.

In the driver, color key parameters appear in the form of D3DTSS_TEXTURECOLORKEYVAL or D3DTSS_DISABLETEXTURECOLORKEY values as arguments to the PFND3DDDI_SETTEXTURESTAGESTATE function. Surprisingly, this is even documented by both the function and it's associated struct argument type: https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/d3dumddi/nc-d3dumddi-pfnd3dddi_settexturestagestate https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/d3dumddi/ns-d3dumddi-_d3dddiarg_texturestagestate

D3DDDIRS_COLORKEYENABLE is a possible argument to PFND3DDDI_SETRENDERSTATE.

So basically you need to track changes to these parameters and override the value of D3DDDIRS_COLORKEYENABLE as per the above logic.

Obviously this isn't bulletproof either. In theory, since there are multiple texture stages, some of them could have color keys while others don't. If those others happen to have black colors in them, then this won't help. But I've not yet seen such a case.

There are of course other ideas to solve the issue, but they are likely computationally more expensive. For example, you could make sure there are no black pixels in any of the textures (replace them with something close in color), and if the game sets black as the color key, replace that with the same replacement color.

Alternatively, make sure all textures have a color key. If they're not supposed to have a color key, then pick a color that's not present in the texture. I guess this could be preconfigured for more efficiency if you know for sure that a game doesn't use a specific color in any of its textures.

[narzoul]

Btw this particular NVIDIA bug was already discussed at length in another topic, here's a short summary post: https://github.com/narzoul/DDrawCompat/issues/116#issuecomment-946084312

This NVIDIA document is also interesting, it gives some insight into their color keying methods (though much of it is probably obsolete by now): https://developer.download.nvidia.com/assets/gamedev/docs/ColorKey.pdf

I saw that on DxWnd forums you also discussed about another color key issue, which causes bilinear filtering to remove too many pixels from color keyed textures. The above document seems to suggest that it may be possible to alter this threshold by enabling alpha testing, and setting the alpha test reference value manually. But I'm pretty sure I already tried this and it had no effect, unfortunately. So to fix this, I had to resort to much uglier hacks, basically editing the pixel shader code generated by ddraw and performing my own configurable "alpha testing" there on each texture lookup operation. As a bonus, at least this also works on D3D9On12, which doesn't support texture color keying at all.

[ghotik]

Thank you very much for the very detailed information and suggestions. I was intrigued by the black pixels remapping solution, a method that I already tried with a Spongebob game to fix the transparent eyes, so I gave it a try. Unfortunately, for the "Rainbow six" game I got a very curious, unexpected and unfortunate "incident": look at the screenshots without and with the pixel color remapping, where (to make the effect more visible, I used a full blue color instead of the almost-black). I was puzzled by the extra text spacing, then I understood: the letters are textures of fixed size with a black right side that delimits the variable width. By remapping the black color the game loses the information to calculate the character width and makes it fixed-spaced at the maximum width. I posted this just in case you are curious. I will try your other suggested methods ...

[narzoul]

Closing for now to clean up stale issues.