Aliasing violation causes erroneous output in GCC 4.4.4

[GoogleCodeExporter]

A user encountered a texture quality problem, in our code that uses the 
Quality_Fastest mode, when compiled with GCC 4.4.4 on Ubuntu 10.10. I 
reproduced it myself with 4.4.4 on Gentoo; the problem didn't occur with 4.3.4 
or 4.5.0.

Eventually I narrowed it down to QuickCompressDXT.cpp selectDiagonal() 
returning non-deterministic results (which was surprising). It uses 
Vector3::xy(), which is defined as:

  inline const Vector2 & Vector3::xy() const
  {
    return *(Vector2 *)this;
  }

which is a clear violation of aliasing rules.

(Unfortunately NVTT seems to compile with no warnings enabled, so GCC wasn't 
complaining "Vector.h:153: warning: dereferencing pointer ‘<anonymous>’ 
does break strict-aliasing rules" like it normally would with "-Wall".)

Fixing this to return a proper non-reference Vector2(m_x, m_y) results in 
correct behaviour. Compilers are good at inlining so this won't be less 
efficient. The other functions that do similar tricks ought to be fixed too.

Original issue reported on code.google.com by exc...@gmail.com on 6 Oct 2010 at 1:59

[GoogleCodeExporter]

This issue was closed by revision r1158.

Original comment by cast...@gmail.com on 6 Oct 2010 at 2:56

• Changed state: Fixed

[GoogleCodeExporter]

Thanks for pointing this out. I've fixed this in trunk and will merge the fix 
in the 2.0 branch shortly.

Original comment by cast...@gmail.com on 6 Oct 2010 at 3:02

• Changed state: Started
• Added labels: Type-Defect

[GoogleCodeExporter]

This issue was closed by revision r1167.

Original comment by cast...@gmail.com on 21 Oct 2010 at 6:54

• Changed state: Fixed