search

llvm / llvm-project

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.
http://llvm.org
Other
27.84k stars 11.47k forks source link

std::numeric_limits<long double>::max_digits10 massively wrong for ppc #26820

Open ec04fc15-fa35-46f2-80e1-5d271f2ef708 opened 8 years ago

ec04fc15-fa35-46f2-80e1-5d271f2ef708 commented 8 years ago
Bugzilla Link 26446
Version unspecified
OS Linux
CC @hfinkel,@hubert-reinterpretcast,@mclow,@nemanjai

Extended Description

libc++ thinks that std::numeric_limits::max_digits10 is 33 for ppc64-linux. However, both DBL_MAX + DBL_DENORM_MIN and DBL_MAX + DBL_DENORM_MIN*2 can be exactly represented by a PPC double double, and you need around 633 decimal digits to distinguish those two values.

hubert-reinterpretcast commented 8 years ago

The precision of PPCDoubleDouble being variable, the guidance given by the C Standard for is much less useful.

For the purposes of determining LDBL_EPSILON, the definition given in prose would set "p" to be 1075; however, IBM's compilers on AIX have traditionally used 106 for "p" for LDBL_EPSILON (since that is a more "conservative" value of p in some contexts).

It appears that the current value of 33 is derived from p = 106. For LDBL_DECIMAL_DIG, the definition in prose would suggest that "p" is 2098.

It seems clear that the most useful value for "p" differs between the various constants (indeed, for LDBL_MAX, the definition in prose would suggest a value of "p" that is not an integer); however, I am not certain that, for each particular constant, there is necessarily a value which is clearly "most useful".