Rounding problem

[tho-ma]

%g does not round the values 5307575.0f or 1104515.0f correctly. Result 5.30757e+6 but should be 5.30758e+6

[MaJerle]

I have commited new changes to develop branch that seem to work properly. When calculating decimal and integer parts of double numbers, 0.5 difference (remaining, for check for rounding) was passing statement as < 0.5f and I have fixed this by checking the difference against 0.5f with small epsilon.

Let's see if it works, it passed my tests for the moment.

[tho-ma]

I still get rounding errors for values like 1022265.0 As a workaround I added a small offset at line 676 in_num += 0.000000000000005;

Additional a zero value ends up in a loop. To correct this the loop at line 672 could be changed to (Checking in_num to be > 0) for (exp_cnt = 0; in_num < 1 && in_num > 0; in_num *= 10, --exp_cnt) {}

There is small typo in lwprintf.h Line 107 should be changed to

define lwprintf_sprintf_ex(lw, s, format, ...) lwprintf_snprintf_ex((lw), (s), SIZE_MAX, (format), ## __VA_ARGS__)

[MaJerle]

I still get rounding errors for values like 1022265.0 As a workaround I added a small offset at line 676 in_num += 0.000000000000005;

Additional a zero value ends up in a loop. To correct this the loop at line 672 could be changed to (Checking in_num to be > 0) for (exp_cnt = 0; in_num < 1 && in_num > 0; in_num *= 10, --exp_cnt) {}

This has been fixed for the moment and it seems to work. Please check or post list of numbers where you see wrong data.

There is small typo in lwprintf.h Line 107 should be changed to

define lwprintf_sprintf_ex(lw, s, format, ...) lwprintf_snprintf_ex((lw), (s), SIZE_MAX, (format), ## VA_ARGS)

Not sure I catch this point?

[tho-ma]

Hi, this approach will not work. Values like 0.44999999 with format %.1g will fail.

[MaJerle]

If we add num += 0.000000000000005; in the prv_calculate_dbl_num_data just before calculating integer and decimal parts, does it work properly for you then?

[tho-ma]

Hi, that will do it for me. There are still some values that can fail but the error is much smaller. The problem seems to be on line 670 and following when dividing or multiplying the value by 10. That may introduce small errors which may have big effects. Therefore, I was thinking of passing the exponent and orig_num to prv_calculate_dbl_num_data and then calculating the difference.

[tho-ma]

By the way this one of the best solution I found. Also the documentation is great. Many thanks