blue channel missing, and red and green channels reversed, when decompressing 3DC compressed normal maps

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Open any 3DC compressed normal map.
2. Decompress with nvdecompress.
3. Open it in any image editor.

What is the expected output? What do you see instead?

I expect that the image includes a blue channel for the Z-component. But
the blue channel is missing. Comparing the decompressed picture with dds
previewers such as WTV and ddsview also shows that the decompressed image
is wrong.

What version of the product are you using? On what operating system?

Latest svn (489) on Fedora 8.

Please provide any additional information below.

Attached are the patch, a test dds file, and two screenshots of the dds
file (one from WTV and one from ddsview). Note that in the patch, I
additionally had to change the X and Y mapping: I had to map X to the green
channel and Y to the red channel for the decompressed texture to be the
same as the previews. Obviously something else is wrong (the 3DC specs
clearly state that the X component should be in the red channel and Y in
the green... ???) In any case with this patch, the decompression seems to
give the same results as can be seen from the other 3rd party programs.

Original issue reported on code.google.com by amorilia...@gmail.com on 26 Mar 2008 at 11:42

Attachments:

• nvtt_3Dc_bchannel_patch.diff
• rib01_triple_ddn.dds
• rib01_triple_ddn_wtv_screenshot.png
• ddsview_screenshot.png

[GoogleCodeExporter]

amorilia,

thanks for the detailed bug report. 

nvdecompress uses a flag in the DDS header to determine whether the ATI2 blocks 
are
used for normal map compression or not. If the file was generated by nvtt the 
flag
should be set, if it was generated by another tool, it's probably not.

The code that handles that is in nvtt/DirectDrawSurface.cpp:903

// If normal flag set, convert to normal.
if (header.pf.flags & DDPF_NORMAL)
{
    if (header.pf.fourcc == FOURCC_ATI2)
    {
        for (int i = 0; i < 16; i++)
        {
            Color32 & c = rgba->color(i);
            c = buildNormal(c.r, c.g);
        }
    }
    ...

ATI2 files can be used for other purposes other than storing normals, so by 
default
the XY blocks are decompressed and stored in the RG channels, just like the 
hardware
does. When the normal flag is present, the normal is reconstructed from the RG 
channels.

It might be interesting to add a command line option to indicate that the input 
is
supposed to be a normal. I think the current behavior should be preserved, 
unless the
general consensus differed. I'll keep the bug open for a while, in case anybody 
wants
to add their 2 cents. Let me know if you have other thoughts.

Original comment by cast...@gmail.com on 27 Mar 2008 at 12:12

[GoogleCodeExporter]

Yes, I hadn't noticed the normal flag check in the code! Indeed it is missing 
on the
particular files I was testing.

So I tested this again by forcing the normal flag on read for ATI2 files:

--- src/nvimage/DirectDrawSurface.cpp   (revision 493)
+++ src/nvimage/DirectDrawSurface.cpp   (working copy)
@@ -604,6 +604,7 @@
        {
                (*stream) << header;
        }
+       if (header.pf.fourcc == FOURCC_ATI2) header.setNormalFlag(true); 
 }

Now running nvdecompress on the file again, I do get a blue channel. I guess it 
does
make sense to assume a normal map by default for ATI2 files even if their flag 
isn't
set, since that's what they have been designed for after all?

Still, one issue left: red and green channels seem swapped in the screenshots 
from
WTV and ddsview. So I downloaded ATI's compressonator, and guess what (see 
attached
screenshot)? :-) Seems like WTV and ddsview don't agree with the 
compressonator. That
leaves me wondering whether it is possible that some ATI2 files have X and Y 
channels
swapped.

Original comment by amorilia...@gmail.com on 27 Mar 2008 at 10:37

Attachments:

• compressonator_screenshot.png

[GoogleCodeExporter]

Oops, scrap that last line, the shot is from test.dds which was one of my tests 
with
nvcompress. Actually compressonator, WTV, and ddsview all agree on the file!!!
Attached a correct screenshot. So it seems that red and green channels must be
swapped after all (how weird nothing of this is mentioned in the specs...).

Original comment by amorilia...@gmail.com on 27 Mar 2008 at 10:41

Attachments:

• compressonator_screenshot.png

[GoogleCodeExporter]

Here's a quick patch for nvdecompress input options, including forcing of the 
normal
map flag.

I still haven't identified the cause of the red and green channels being 
swapped on
3DC. Any progress on that front?

Original comment by amorilia...@gmail.com on 6 Apr 2008 at 9:22

Attachments:

• nvdecompressoptions.patch

[GoogleCodeExporter]

Sorry for the delay, I'll integrate this ASAP. I thought that in comment 3 you
retired what you said about about the swapped channels. Let me check that out.

Original comment by cast...@gmail.com on 11 Apr 2008 at 9:11

• Changed state: Accepted

[GoogleCodeExporter]

I've integrated your patch. I also looked at the problem of the swapped 
coordinates.
If you look at the DX10 block compression format spec:

http://msdn2.microsoft.com/en-us/library/bb694531.aspx

you will see that BC5 is supposed to have two blocks, the first one encodes the 
R and
the second one encodes the G channels. That's what I implement in the texture 
tools.
However, ATI's 3Dc format uses the opposite convention, G and then R.

Unfortunately, most viewers out there assume the 3Dc convention. On the other 
side,
the compressonator uses the bitcount field of the DDS header to indicate how the
components are swizzled. However, I haven't been able to figure out how's that 
done.
Here are the bitcount values for some of their formats:

ATI2 YX:            0 (0x00000000)
ATI2 XY:   1498952257 (0x59583241) (BC5)
DXT5 ATI2:  893661761 (0x35443241)
DXT5 xGBR: 1380075384 (0x52424778)
DXT5 RxBG: 1195538514 (0x47427852)
DXT5 RGxB: 1199063634 (0x47784252)
DXT5 xRBG: 1195528824 (0x47425278)
DXT5 RGxB: 1115178834 (0x42784752)
DXT5 xGxR: 1383614328 (0x52784778) (DXT5n)

Any idea how the swizzle is encoded?

Original comment by cast...@gmail.com on 11 Apr 2008 at 10:58

[GoogleCodeExporter]

Thanks for integrating the patch!

I'm not sure what your question is about the swizzle. You mean to ask whether 
there's
a pattern linking the bitcount field to the swizzle in some logical way? The 
bitcount
field is simply used as a string as far as I can see. For instance, the first is

0x59583241 = 0x41 0x32 0x58 0x59 = "A2XY"

and the last is

0x52784778 = 0x78 0x47 0x78 0x52 "xGxR"

and so on (the second expression is how it's stored in the memory: the byte 
order
must be swapped to read the integer as a 4-byte string).

Original comment by amorilia...@gmail.com on 13 Apr 2008 at 4:11

[GoogleCodeExporter]

Doh! that was so obvious. Those are simply FOURCC codes. I'll change the 
compressor
to output the files with the appropriate swizzle code, and improve the 
decompressor
to handle most of them.

Original comment by cast...@gmail.com on 14 Apr 2008 at 8:23

[GoogleCodeExporter]

Brilliant! I'll be looking forward to the implementation, as it would be nice to
support all those formats with Blender too. Aside, for your information, your 
DDS
decompression code, which I have ported to Blender, will likely make it 
(finally!)
into the next release, Blender 2.46. :-)

Original comment by amorilia...@gmail.com on 14 Apr 2008 at 10:46

[GoogleCodeExporter]

That is awesome!

ATI2 files are not output with the appropriate swizzle code, so they should be
displayed correctly in third party tools.

I'll add support for reading the swizzle codes next.

Original comment by cast...@gmail.com on 17 Apr 2008 at 7:19

[GoogleCodeExporter]

Original comment by cast...@gmail.com on 30 Apr 2008 at 6:57

• Added labels: Milestone-Release2.1

[GoogleCodeExporter]

Original comment by cast...@gmail.com on 11 May 2008 at 7:59

• Added labels: Milestone-Release2.2
• Removed labels: Milestone-Release2.1

[GoogleCodeExporter]

I check out the latest version from svn, and "red and green channels reversed" 
the problem still exists.

how to work around this?

Original comment by wuyu....@gmail.com on 11 Aug 2011 at 8:51

[GoogleCodeExporter]

finally, manually swap the channel fixed this issue. 

ugly solution, hope for a better one.

Original comment by wuyu....@gmail.com on 23 Aug 2011 at 6:51