search

JonBCode / openjpeg

Automatically exported from code.google.com/p/openjpeg
Other
0 stars 0 forks source link

Inconsistent fixed point rounding #489

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
static INLINE OPJ_INT32 opj_int_fix_mul(OPJ_INT32 a, OPJ_INT32 b) {
    OPJ_INT64 temp = (OPJ_INT64) a * (OPJ_INT64) b ;
    temp += 4096;
    assert((temp >> 13) <= (OPJ_INT64)0x7FFFFFFF);
    assert((temp >> 13) >= (-(OPJ_INT64)0x7FFFFFFF - (OPJ_INT64)1));
    return (OPJ_INT32) (temp >> 13);
}

The above code will round 1.5f up to 2.0f, and -1.5f up to -1.0f.

(when I use floats, I mean their fixed point equivalent)

Original issue reported on code.google.com by boxe...@gmail.com on 30 Apr 2015 at 2:38