ColorKeyMethod using alphatest

[elishacloud]

How are you converting color keyed textures to an ARGB format with the alpha channel enabled for the color? Are you creating a new surface/texture and then copying the surface data over?

Does this work for both Bltting with color keys and when color keys are using as a render state: D3DRENDERSTATE_COLORKEYENABLE?

I am just asking because I am working on a wrapper that converts Direct3D7 to Direct3D9 and color keying is not supported in Direct3D9 so I am using a shader for color keys and I am using either D3DX or my own emulation for Bltting.

[elishacloud]

I found the code check-in for this here. Looks like you are just using a pixel shader for color keying. It looks like you are talking to the D3dDdi driver directly to set the pixel shader. I am trying to avoid talking to the driver and just use Direct3D9 public functions.

[narzoul]

How are you converting color keyed textures to an ARGB format with the alpha channel enabled for the color? Are you creating a new surface/texture and then copying the surface data over?

Yes, exactly. The original texture is rendered into an ARGB RTT (render target texture) with a shader that sets the alpha to 0.0 for color keyed texels, and 1.0 for everything else.

Does this work for both Bltting with color keys and when color keys are using as a render state: D3DRENDERSTATE_COLORKEYENABLE?

I'm not sure I understand the question, but D3DRENDERSTATE_COLORKEYENABLE does not affect blitting, if that's what you are asking.

It looks like you are talking to the D3dDdi driver directly to set the pixel shader. I am trying to avoid talking to the driver and just use Direct3D9 public functions.

I'm forced to do that, because the DX7 runtime doesn't support shaders. Of course, you don't need to do that if you can use the DX9 runtime to handle your shaders.

[elishacloud]

I see. Then you are not doing anything for the color key used by IDirectSurface::Blt()?

I am using a shader to just ignore the pixels that are used by the high/low color key. See here.

How do you handle color spaces?

[narzoul]

I see. Then you are not doing anything for the color key used by IDirectSurface::Blt()?

Those are handled by Resource::shaderBlt() -> ShaderBlitter::textureBlt() -> ColorKey.hlsl

No special preparation, other than copying the source surface to a texture if it's not already in a texture resource (to allow sampling from the shader), and similarly to get a temporary render target if the destination surface is not suitable. Probably doesn't work well with BltFilter=linear if stretching is used, but I haven't seen an example app with stretched color keyed blitting yet.

I am using a shader to just ignore the pixels that are used by the high/low color key.

I don't think any driver ever supported color key ranges, you can probably just assume that the low and high values are the same. You can't even pass a range to WDDM drivers, nor can you use both a source color key and a destination color key at the same time. I've never seen an app using destination color keying though.

How do you handle color spaces?

Assuming you mean YUV formats, I don't have any special handling for them. I let native ddraw and the driver's Blt routine take care of it. AFAIK the driver's Blt routine should support a simple copy from YUV FOURCCs to RGB formats. Not sure if it supports stretching or color keying, but it's probably not used by anything in practice anyway. These formats are generally only used by some movies, so the only operation ever done with them is converting them from YUV to RGB format with Blt.

Possibly useful material about texture color keying: https://developer.download.nvidia.com/assets/gamedev/docs/ColorKey.pdf

[elishacloud]

Thanks for your comments.

By color space I meant the values from the low color key to the high color key DDCKEY_COLORSPACE, see here.

As far as low/high color keys, you might be right that no hardware supported it. I have never seen a game use both (meaning a different value for low and high). I wonder why it's still in the Microsoft documentation if it does not work: https://learn.microsoft.com/en-us/windows/win32/api/ddraw/ns-ddraw-ddcolorkey

I find these old functions very confusing and the documentation very sparse and not always correct.

Possibly useful material about texture color keying: https://developer.download.nvidia.com/assets/gamedev/docs/ColorKey.pdf

Thanks for the link!

[elishacloud]

Interesting. On my hardware if I set the DDCKEY_COLORSPACE flag and set low and high values differently then SetColorKey() fails with DDERR_NOCOLORKEYHW and a Blt() that uses color key fails with DDERR_INVALIDPARAMS.

[elishacloud]

Thanks for your help here. I am closing this issue.