Colorkey query: rendering on CPU?

[BEENNath58]

I was playing POD and I was shocked at how bad the Nvidia color keys are in the game, there were holes everywhere and there. I haven't tried DDC or any other wrapper yet, but was more interested to know if it's possible to push the color keys to be rendered via the CPU.

Is it doable? Would there be any visual changes in the rendering if it's done? Is it possible for both DirectDraw as well as Direct3D methods?

[narzoul]

DirectDraw can do color keying via HEL, yes. It's automatically used in every situation where hardware color keying is not supported, e.g. blitting between system and video memory, or between two system memory surfaces, or if the driver reports no support for hardware color keying. The HEL always uses point filtering when stretching, so as long as the driver also uses point filtering, in theory the results should be similar, although rounding errors may lead to different pixels getting duplicated/discarded. I seem to recall that NVIDIA drivers also produce visible 1-pixel shifts when stretching, even without color keying.

Direct3D only supports texture color keying when using a software device, e.g. the RGB device. There is no fallback to software for HAL devices.

[BEENNath58]

I am more interested about the situation where hardware colorkeying is obviously available but just SW caps are used because HW abstracted ones are bugged. Specifically I recall somewhere you used the same technique for AMD RX 5700. If I understand it correct, what you did for this card is the entire DD and D3D rendering is on the GPU (except the colorkey caps and a few others...)

[narzoul]

I don't understand what you mean.

I think there were two fixes for color keyed blits affecting the RX 5xxx series. The first one simply moved the blits to software. I may have used my own software blitter for this already, or just relied on HEL by clearing the colorkey support caps in the driver, I don't remember which one I used.

The other (current) fix is that color keyed blits are converted to Direct3D rendered quads with texture color keying.

For Direct3D, the only fix for texture color keying was disabling D3DRENDERSTATE_COLORKEYENABLE when none of the texture stages use color keying. This was a workaround for NVIDIA using black as color key when a texture has no color key assigned.

[BEENNath58]

I think there were two fixes for color keyed blits affecting the RX 5xxx series. The first one simply moved the blits to software. I may have used my own software blitter for this already, or just relied on HEL by clearing the colorkey support caps in the driver, I don't remember which one I used.

For the 2nd one (clearing HAL support caps), does this present a correct and proper look as it would be in HW mode (for example, if I put the whole application in EMULATIONONLY mode I will have no bilinear filtering; will "just" HEL colorkeys have such a tradeoff with another feature?)

For Direct3D, the only fix for texture color keying was disabling D3DRENDERSTATE_COLORKEYENABLE when none of the texture stages use color keying. This was a workaround for NVIDIA using black as color key when a texture has no color key assigned.

I recall in Interstate 76, the cad textures would be transparent in a lot of places that seemed black. Was this in operation at that spot? Is this the correct texture specification, like does lack of color keying = black?

Btw is there a colorkey state enabler/disabler for textures with DD? Such as you have COLORKEYENABLE in D3D as true/false, is there one for DD?

[narzoul]

For the 2nd one (clearing HAL support caps), does this present a correct and proper look as it would be in HW mode (for example, if I put the whole application in EMULATIONONLY mode I will have no bilinear filtering; will "just" HEL colorkeys have such a tradeoff with another feature?)

What is "correct and proper look as it would be in HW mode"? How the hardware does color keying is implementation dependent, at least when stretching is also involved, since the stretching algorithm itself is up to the driver. That includes how stretching interworks with color keys. If no stretching is involved, then the correct result should be pretty straightforward.

With the HEL software blitter, you'll get the same results as with EMULATIONONLY, since the latter just forces the HEL to be used for every DirectDraw operation, even if the hardware could accelerate it.

I recall in Interstate 76, the cad textures would be transparent in a lot of places that seemed black. Was this in operation at that spot?

Probably, but I don't recall testing that particular game.

Is this the correct texture specification, like does lack of color keying = black?

No, this is an NVIDIA bug.

Btw is there a colorkey state enabler/disabler for textures with DD? Such as you have COLORKEYENABLE in D3D as true/false, is there one for DD?

Color keying is never implicitly enabled for DirectDraw, even if a surface has an attached color key. You have to explicitly request it by passing the appropriate flags to Blt (e.g. DDBLT_KEYSRC).

[BEENNath58]

What is "correct and proper look as it would be in HW mode"?

Let's say, DDC forces DirectDraw ColorKey caps in DX Caps Viewer set to NO for Primary Display Driver. But the one under Hardware Emulation Layer is set YES.

Thus, is "rendering Colorkeys on Cpu" the only difference that happens when a program is run on DDraw?

As a result, are there a few more other DDraw capabilities that change to processing on CPU (which in turn changes the graphical output) ?

[narzoul]

Only those blits will be performed by HEL that are unsupported by HAL, i.e. color keyed blits in this case, plus whatever else was already unsupported. Everything else will still be done by HAL. This is the default behavior for every DirectDraw blit feature.

Note that the entire blit is done by HEL in this case, including the stretching of the non-transparent parts. There is also a performance impact from having to lock and then unlock both surfaces for CPU access for the duration of the blit. If done frequently, this will likely kill performance. It may be faster to just do everything on CPU, with no video memory involved.

[BEENNath58]

Only those blits will be performed by HEL that are unsupported by HAL, i.e. color keyed blits in this case, plus whatever else was already unsupported. Everything else will still be done by HAL. This is the default behavior for every DirectDraw blit feature.

Note that the entire blit is done by HEL in this case, including the stretching of the non-transparent parts. There is also a performance impact from having to lock and then unlock both surfaces for CPU access for the duration of the blit. If done frequently, this will likely kill performance. It may be faster to just do everything on CPU, with no video memory involved.

Ok so the summary is this: DirectDraw HEL can do color keying on its own and to expect good performance, surface can be drawn in System Memory.

And the only Direct3D problematic part would be with Nvidia, where you get black color at a place where no color key is defined. And the fix would be to disable the parameter you mentioned. Is there any app/game to test this black color issue?

[narzoul]

Ok so the summary is this: DirectDraw HEL can do color keying on its own and to expect good performance, surface can be drawn in System Memory.

Created in system memory, but yes. But this doesn't work if the surface is also used as a Direct3D HAL texture or render target.

And the only Direct3D problematic part would be with Nvidia

And D3D9On12, which doesn't support texture color keying at all.

Nvidia, where you get black color at a place where no color key is defined

No, you get transparency where the texture is black, even though no color key is set for the texture.

And the fix would be to disable the parameter you mentioned.

Yes.

Is there any app/game to test this black color issue?

Mig Alley.

[BEENNath58]

Created in system memory, but yes. But this doesn't work if the surface is also used as a Direct3D HAL texture or render target.

How do you check a texture is a D3D HAL texture or a RTT?

No, you get transparency where the texture is black, even though no color key is set for the texture.

Okay tested and that happens in Interstate 76 with -d3d parameter. 👍 I get a confusion with Rainbow Six 1998. In the game, if the D3DSETRENDERSTATE_COLORKEYENABLE is returned FALSE, the transparent textures in the compound buildings correctly get colored to black, but the grills get blackened as well. What is the case here?

For example: the fixed textures look like this-

And something break like-

With the COLORKEYENABLE TRUE on a Intel HW, the green grass (goodtex.png) is rendered wholly and the plant (badtextures.png) is rendered with transparency around. Same happens with DDC. If I give COLORKEYENABLE FALSE value, the green plant with holes is fixed while the plants get a black background. So how does a Intel/AMD HW interpret non-colorkeyed textures, based on th behaviour of green grass and plant?

If I use DDC here, both of them work correctly. Why? Does DDC do anything special for this situation? Is it based on shader and if so did it get fixed before using this earlier?

Mig Alley.

Ah is there some other game? Mig Alley has a big box issue on Nvidia, and DDC fixes the entire game so I can't check the issue

[narzoul]

I get a confusion with Rainbow Six 1998. In the game, if the D3DSETRENDERSTATE_COLORKEYENABLE is returned FALSE, the transparent textures in the compound buildings correctly get colored to black, but the grills get blackened as well. What is the case here?

Some textures are supposed to be transparent at the black pixels, but you turn off transparency for those too.

With the COLORKEYENABLE TRUE on a Intel HW, the green grass (goodtex.png) is rendered wholly and the plant (badtextures.png) is rendered with transparency around. Same happens with DDC. If I give COLORKEYENABLE FALSE value, the green plant with holes is fixed while the plants get a black background. So how does a Intel/AMD HW interpret non-colorkeyed textures, based on th behaviour of green grass and plant?

I tried for several minutes to interpret what you're saying here, but I'm not sure I understand what you're saying. Goodtex.png, badtextures.png, are these supposed to be the pics above? They don't have names. They're also so dark I can hardly see anything, I had to adjust the colors with an image editor just to tell what I'm looking at. Green grass, plant, green plant, what are you referring to with these, and in which picture? Where are the holes supposed to be in "green plant with holes"?

I don't recall having any transparency issues on my old Intel iGPU, but I don't have it anymore, so I can't retest it.

If I use DDC here, both of them work correctly. Why? Does DDC do anything special for this situation? Is it based on shader and if so did it get fixed before using this earlier?

The only fix I can recall is the dynamic switching of D3DRENDERSTATE_COLORKEYENABLE, depending on whether the currently used textures have color keys attached or not. You seem to be doing it in an always on/always off way, which won't work, since you're disabling transparency for everything, even for objects that should be transparent.

Ah is there some other game? Mig Alley has a big box issue on Nvidia, and DDC fixes the entire game so I can't check the issue

Viper Racing Demo. At the start of the race, the poles holding the START flag are missing, as well as the bottom of the car (can be seen through the windows).

[BEENNath58]

I tried for several minutes to interpret what you're saying here, but I'm not sure I understand what you're saying. Goodtex.png, badtextures.png, are these supposed to be the pics above? They don't have names. They're also so dark I can hardly see anything

It was fully visible here, idk maybe its OLED. I noticed I added the files directly not as attachment, sorry for that.

Where are the holes supposed to be in "green plant with holes"?

Around the black region. I got your word from the following part:

The only fix I can recall is the dynamic switching of D3DRENDERSTATE_COLORKEYENABLE, depending on whether the currently used textures have color keys attached or not. You seem to be doing it in an always on/always off way, which won't work, since you're disabling transparency for everything, even for objects that should be transparent.

So I get this: the setting must be FALSE for non-color keyed surfaces. Surfaces with color keys should not be altered by our setting.

Edit: Just remembered there's a white -> magenta color transparency issue on Nvidia too. What was the issue there? I recall this issue in Spongebob: Employees of the Month