search

nagyistoce / stringencoders

Automatically exported from code.google.com/p/stringencoders
Other
0 stars 0 forks source link

In some cases modp_dtoa/modp_dtoa2 gives different precision then requested #12

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
Nick,

The code has the following branch:

    if (value > thres_max) {
        sprintf(str, "%e", neg ? -value : value);
        return;
    }

This is wrong as it ignores requested precision - sprintf uses 6 precision by 
default.

Original issue reported on code.google.com by ilya.m...@gmail.com on 10 Jun 2010 at 8:56

GoogleCodeExporter commented 9 years ago 
I think what is going on here is that,

"if the precision is beyond what we can print, use exponential notation"

This may be stupid. and probably should be removed so the output is consistent.

--nickg

Original comment by nickgsup...@gmail.com on 12 Jun 2010 at 2:10