Bug: Precision is effecting whole value with numbers larger than precision value

[mikeburgh]

If you enter the value 93454555 then the output will be 93,454,600

So it seems it's applying to the whole value, not just the decimal side

A temp fix is to set the precision setting to 16, but it still applies to the whole number, and not just the part after the decimal

[danilort]

It is a visualization problem, which does not concern the calculations. A temp workaround is to use format().

[danilort]

FROM: https://mathjs.org/docs/reference/functions/format.html

precision: number Limit the number of digits of the formatted value. 

In case of notations ‘exponential’, ‘engineering’, and ‘auto’, 
precision defines the total number of significant digits returned. 

In case of notation ‘fixed’, 
precision defines the number of significant digits after the decimal point. 

We need a new setting to choose the type of notation: engineering, auto, fixed. For example:

"qalc.output.notation": "fixed"

and a new function, to set a different format in each file. For example:

defaultFormat({'notation': 'fixed', 'precision': 2})

[danilort]

Using:

// Set the precision after the decimal of the output
"qalc.output.precision": 2

I always get 3 significant digits. It has no effect on the decimals:

[danilort]

This is the expected result:

[nortakales]

Adding a setting to configure the notation will be really easy.

I'm trying to figure out how best to configure these settings per file though. I think what you suggested is a good idea, something like:

defaultFormat({'notation': 'fixed', 'precision': 2})

I might use setFormat as the function name though, and have it apply to any expressions that come after it (not the entire file). You could set it at the beginning of the file, and then change the format half way through the file if desired.

[nortakales]

I'm introducing two new settings in the next release:

qalc.output.notation - options are the same as for math.js, and this will pass through to the math.js format call. qalc.output.trimTrailingZeros - because what I really intended in the first place was to be able to apply precision only to the decimal value, but not have things like 1.000. That doesn't seem possible with math.js natively so this setting can be used with fixed notation to round only the decimal place AND trim off those extra zeroes.

For the ability to change formats within a file using a new function, I opened an issue specifically for that: https://github.com/nortakales/vs-code-qalc/issues/10

[nortakales]

Bug fixed in Qalc v0.1.8