error in OptimalCompress::compressDXT1G

[GoogleCodeExporter]

What steps will reproduce the problem?
1. compress a flat normalmap as DXT5n (128,128,255)

What is the expected output? What do you see instead?
I expect a texture with rgba=(255,128,0,128). 
I see a texture with rgba=(255,0,0,127).

What version of the product are you using? On what operating system?
last svn

Please provide any additional information below.
OptimalCompress::compressDXT1G takes as first argument rgba.color(0).
It should take rgba.color(0).g instead.
The obtained dds has now a flat value of (255,126,0,127).
Is there a better way to correct this behaviour and obtain a better quality
texture (since "flat" normal is an important normal)?

Original issue reported on code.google.com by marcoalt...@gmail.com on 11 Mar 2009 at 2:52

[GoogleCodeExporter]

The orinal normal (128,128,255) is normalized by the initial renormalization 
process
to (127,127,255). Is it correct?

Original comment by marcoalt...@gmail.com on 11 Mar 2009 at 2:54

[GoogleCodeExporter]

I cannot reproduce the behavior you describe. I get a texture with the following
values: (255, 125, 0, 127), the texture is decoded as follows:

X = R * A
Y = G
Z = sqrt(1 - X^2 - Y^2)

In this case that produces:

X = 127
Y = 125
Z = 254

which is exactly what nvdecompress produces. You don't get exactly the same 
input
because DXT is a lossy compression format and introduces quantization errors.

compressDXT1G correctly uses the green component of the first color of the 
block. See
OptimalCompressDXT.cpp line 310:

uint8 singleColor = rgba.color(0).g;

The code in the 2.0 branch does not have a DXT5n single color compressor, but
produces the same result anyway, it's just a little bit slower.

Closing as invalid. Please, reopen if you think I may be missing something and 
you
want to provide additional information. Thanks!

Original comment by cast...@gmail.com on 18 Mar 2009 at 6:50

• Changed state: Invalid

[GoogleCodeExporter]

I'm sorry: I wrongly decommented lines 412-416 in compressdxt.cpp. This code 
executes
OptimalCompress::compressDXT1G at line 283, so the rgba.color(0) is converted to
uint32 and then clamped to uint8.

Original comment by marcoalt...@gmail.com on 18 Mar 2009 at 9:05