modp_dtoa may exceed precision due to rounding

[GoogleCodeExporter]

If you convert certain fractions - say 0.09999999 to precision 6 - the code 
will round up to 0.1000000 (999999 is the fraction part, plus 1 for the 
rounding). This produces 7 digits past the decimal.

Looks like modp_dtoa2 handles this indirectly, but it might make sense to check 
if the rounding causes a carry, and if so, to divide by 10.

Original issue reported on code.google.com by bokel...@gmail.com on 29 Dec 2010 at 4:38

[GoogleCodeExporter]

ok I'll take a look "this year"

Original comment by nickg@client9.com on 29 Dec 2010 at 7:45

[GoogleCodeExporter]

Hi,

I just checked in some enhancements to the tests to try and catch this.   For 
better or worse I can't replicate this.

Among other this I added this:

static char* testRoundingPrecisionOverflow() {
    char buf1[100];

    modp_dtoa(0.09999999, buf1, 6);
    mu_assert_str_equals(buf1, "0.100000");
    modp_dtoa2(0.09999999, buf1, 6);
    mu_assert_str_equals(buf1, "0.1");
    return 0;
}

if you 'make test' does this pop?

-ffastmath might be culprit

nickg

Original comment by nickgsup...@gmail.com on 3 Jan 2011 at 1:06

[GoogleCodeExporter]

Nope, tests work fine on svn trunk:

[bokelley@sandbox stringencoders-read-only]$ make test
test/modp_b16_test.c ........OK (8 tests)
test/modp_b64_test.c .....OK (5 tests)
test/modp_b64w_test.c .....OK (5 tests)
test/modp_b85_test.c .....OK (5 tests)
test/modp_b2_test.c ........OK (8 tests)
test/modp_burl_test.c ..........OK (10 tests)
test/modp_bjavascript_test.c .....OK (5 tests)
test/modp_numtoa_test.c ...........OK (11 tests)
test/modp_ascii_test.c ........OK (8 tests)

Original comment by bokel...@gmail.com on 4 Jan 2011 at 8:18