Rounding errors when converting hex/bin values to decimal

[AndrzejKowalski9917]

The evaluation of long hex / bin / oct values fails with rounding errors (probably due to float conversion).

Example:

0b11111111111111111110   =1048570 // should be 1048574
0b11111111111111111111   =1048580 // should be 1048575

oct(0b111111111111111111110)  ="0o7777776" // evaluating to octal works
oct(0b111111111111111111111)  ="0o7777777" // evaluating to octal works

0xffffffff               =4294970000 // should be 4294967295

Is there a way to force evaluation as decimal integer value, like dec(0b111111111111111111110)?

[AndrzejKowalski9917]

I found the option "qalc.output.precision" in the settings. Setting it to '16' formats all the output in the issue correctly. If the value exceeds a certain bit length, the output is displayed as exponential number with rounding. As I found out, this is due to java script handling all numbers as ieee-754 float, which limits the precission: see Stackoverflow. To work arround this, one would need to use a library like Big Numbers.

For my needs, the precission is accaptable. Maybe for the future it would be a nice feature to add at least 64 bit decimal output to improve the usage as programmer calculator.

[nortakales]

Part of this issue (the unexpected rounding) should be fixed by https://github.com/nortakales/vs-code-qalc/issues/4 where I've introduced a couple new settings to control output format. Also, the defaults for those settings should provide much more sane rounding out of the box.

I think the other part you mention (64 bit numbers without exponents) is already working if you set the "Upper Exponent Bound" setting higher. I set it to 30 and was able to convert 64 bit binary numbers into decimal without an exponent.

Please reopen if I misunderstood something!

[AndrzejKowalski9917]

Hey,

thanks for the answer and your effort! I'm ok with how it works now.

I just want to clarify what I meant with the 64 bit binary issue.

0xffffffffffffffff     =18446744073709552000 // should be 18446744073709551615

As far as I understood this is due to how Java Script handles all numbers internally (as 64 bit ieee-754 double precision), so you lose precision if the number exceeds a certain amount of bits. There seem to exist libraries to work arround this issue (like the mentioned Big Numbers, but using this would probably exceed the scope of this extension.

To quote the answer on Stackoverflow:

JavaScript represents numbers using IEEE-754 double-precision (64 bit) format. As I understand it this gives you 53 bits precision, or fifteen to sixteen decimal digits. Your number has more digits than JavaScript can cope with, so you end up with an approximation. This isn't really "mishandling" as such, but obviously it isn't very helpful if you need full precision on large numbers. There are a few JS libraries around that can handle larger numbers, e.g., BigNumber and Int64.

This is consistent with the documentation your extension mentions on the Setting Qalc-Output:Precision. From the documentation:

precision: number Limit the number of digits of the formatted value. For regular numbers, must be a number between 0 and 16. For bignumbers, the maximum depends on the configured precision, see function config(). In case of notations ‘exponential’, ‘engineering’, and ‘auto’, precision defines the total number of significant digits returned. In case of notation ‘fixed’, precision defines the number of significant digits after the decimal point. precision is undefined by default.

[nortakales]

I see what you mean now. After looking in to some more Math.js features I found the bignumber() function. I think this might be what you are looking for

0xffffffffffffffff                 = 18,446,744,073,709,552,000
18446744073709551615               = 18,446,744,073,709,552,000

bignumber('18446744073709551615')  = 18,446,744,073,709,551,615
bignumber('0xffffffffffffffff')    = 18,446,744,073,709,551,615

Note: the quotes are necessary!