client9 / stringencoders

Fast c-string transformations
MIT License
135 stars 51 forks source link

In some cases modp_dtoa/modp_dtoa2 gives different precision then requested #12

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago
Nick,

The code has the following branch:

    if (value > thres_max) {
        sprintf(str, "%e", neg ? -value : value);
        return;
    }

This is wrong as it ignores requested precision - sprintf uses 6 precision by 
default.

Original issue reported on code.google.com by ilya.m...@gmail.com on 10 Jun 2010 at 8:56

GoogleCodeExporter commented 9 years ago
I think what is going on here is that,

"if the precision is beyond what we can print, use exponential notation"

This may be stupid. and probably should be removed so the output is consistent.

--nickg

Original comment by nickgsup...@gmail.com on 12 Jun 2010 at 2:10

client9 commented 8 years ago

happy to have pull request to fix this.