josdejong / mathjs

An extensive math library for JavaScript and Node.js
https://mathjs.org
Apache License 2.0
14.4k stars 1.24k forks source link

Formatting numbers without exponential notation (for bignumbers, also)? #676

Closed balagge closed 6 years ago

balagge commented 8 years ago

I have a scenario where I need a string representation of a number without exponential notation (i.e. always in "normal" decimal notation). No rounding should occur, and the actual (highest available) precision value is required. Also, no unnecessary trailing zeros after the decimal point, please :)

Now this is crazily difficult in plain javascript, as far as I know there is no simple solution for that. See this mess:

http://stackoverflow.com/questions/1685680/how-to-avoid-scientific-notation-for-large-numbers-in-javascript

I checked math.format, but, unfortunately, it copies the behavior of javascript Number.prototype.toFixed() , in the sense that specifying notation:"fixed"will always require an actual precision setting (if omitted, defaulted to zero), which is unfortunate, because it causes rounding and/or adding unnecessary 0's after the last significant digit after the decimal point. So, at the end of the day, it returns a different or non-canonical string representation of the input number. By the way, toFixed() does not deliver its promise even in the sense that it returns exponential notation sometimes (???!!). So again, toFixed() is pretty useless, and, by copying its behavior, math.format() also seems pretty useless.

Then I realized that I can solve the problem by using notation:"auto"(which is the default) because it allows the additional option exponential and I can set exponential:{lower:0, upper:Infinity}. This does the job nicely! No rounding, no unnecessary zeroes added!

However, when I use the same function for decimals (bignumbers), a run-time exception occurs.

I checked the source and I found this (lib/utils/bignumber/formatter.js, lines 118-126):

var oldConfig = {
  toExpNeg: value.constructor.toExpNeg,
  toExpPos: value.constructor.toExpPos
};

value.constructor.config({
  toExpNeg: Math.round(Math.log(lower) / Math.LN10),
  toExpPos: Math.round(Math.log(upper) / Math.LN10)
});

This is where the exception occurs, because the value.constructor.toExpNeg and value.constructor.toExpPos do not accept values -Infinitiy and Infinity, respectively (where -Infinity = Math.log(0), and Infinity = Math.log(Infinity).

However, the lines above are not needed at all, since oldConfig is not referenced in the source at all, and setting the toExpNeg and toExpPos properties is not required on the constructor (as far as I see), since the following lines (134-141) check the limits for exponential notation, and it is not decimal.js that decides whether to use exponential or fixed notation.

So I tried removing the above lines, and everything works as expected! Try:

var math = require('mathjs');

var numbers = [
1.1234567890123456789e+30, 
1.1234567890123456789e-30, 
-1.1234567890123456789e+30,
-1.1234567890123456789e-30, 
math.bignumber("0.12345678901234567890123456789012345678901234567890123456789012345678901234567890"), 
math.bignumber("12345678901234567890123456789012345678901234567890123456789012345678901234567890")]

var i;
for (i=0;i<numbers.length;i++) {
    console.log(math.format(numbers[i],{exponential:{lower:0,upper:Infinity}}));
}

with the above lines this produces an exception.

Error: [DecimalError] Invalid argument: toExpNeg: -Infinity

without them, it produces the expected results:

1123456789012345700000000000000
0.0000000000000000000000000000011234567890123457
-1123456789012345700000000000000
-0.0000000000000000000000000000011234567890123457
0.1234567890123456789012345678901234567890123456789012345678901235
12345678901234567890123456789012345678901234567890123456789012350000000000000000

Note: loss of some significant digits is normal, both for the javascript numbers (float) and for the decimals (precision is set to 64 significant digits).

So maybe removing lines 118 through 126 from formatter.js could be a solution? But I kind of hesitate, because of a comment in the source (line 117):

 // adjust the configuration of the BigNumber constructor (yeah, this is quite tricky...)

So maybe I am missing a point here?

Update: checked for complexes as well, the setting {exponential:{lower:0,upper:Infinity}} works fine, also!

Update2: decimal.js does correct the messy behavior of javascript toFixed(), it drops both the (stupid) defaulting of precision to 0, and the (even crazier) exponential notation in the return value. See

http://mikemcl.github.io/decimal.js/#toFixed

Where it reads

" Unlike Number.prototype.toFixed, which returns exponential notation if a number is greater or equal to 10^21, this method will always return normal notation.

If dp is omitted, the return value will be unrounded and in normal notation. This is unlike Number.prototype.toFixed, which returns the value to zero decimal places[...]"

josdejong commented 8 years ago

Do I summarize your story correctly if you would like to be able to define {exponential: {lower: -Infinity, upper: Infinity}}? That makes sense.

The lower and upper configuration expects a value, not an exponent. The following works as expected:

var numbers = [
1.1234567890123456789e+30, 
1.1234567890123456789e-30, 
-1.1234567890123456789e+30,
-1.1234567890123456789e-30, 
math.bignumber("0.12345678901234567890123456789012345678901234567890123456789012345678901234567890"), 
math.bignumber("12345678901234567890123456789012345678901234567890123456789012345678901234567890")]

var i;
for (i=0;i<numbers.length;i++) {
    console.log(math.format(numbers[i],{exponential:{lower:1e-100,upper:1e100}}));
}

outputs:

1123456789012345700000000000000
0.0000000000000000000000000000011234567890123457
-1123456789012345700000000000000
-0.0000000000000000000000000000011234567890123457
0.1234567890123456789012345678901234567890123456789012345678901235
12345678901234567890123456789012345678901234567890123456789012350000000000000000
balagge commented 8 years ago

Yes, but I guess {exponential: {lower: 0, upper: Infinity}} makes more sense as it is the absolute value that is compared to these bounds.

Also, as I said above, the code works as is, provided we remove lines 118-126 from lib/utils/bignumber/formatter.js, so basically the question is: are those lines required at all? They do not seem to make much sense to me.

Sorry for the long post. The reason I write with detail is because I keep forgetting things, so I like to put down everything that has lead to a certain conclusion / request.

balagge commented 8 years ago

Or, another (and, in my opinion, better) approach could be to correct the messy behavior of javascript toFixed() in math.format({notation:"fixed"}) -- as is done in decimal.js.

My point here is that the current behavior of mathjs math.format({notation:"fixed"}) copies the javascript native behavior: (0.1).toFixed() returns "0" in javascript (why? should return "0.1"!) (1e21).toFixed() returns "1e+21" (omg!) This is a mess.

Whereas the implementation in decimaljs makes perfect sense: math.bignumber(0.1).toFixed() returns "0.1", math.bignumber(1e21).toFixed() returns "1000000000000000000000".

balagge commented 8 years ago

I don't get your point with math.format(numbers[i],{exponential:{lower:1e-100,upper:1e100}}). This has a limited range. Of course it works in the given range, but not generally. Especially, as the range is a number, for XL / XS decimals (i.e. larger/smaller than floating-point range) cannot have fixed notation this way.

josdejong commented 8 years ago

Sorry for the confusion, for some reason I thought you passing upper and lower options as a exponent instead of a value. I was a bit lost in the amount of information in your post ;).

It makes sense to me to change the default behavior of math.format(value, {notation:"fixed"}) (with no precision provided) to return all decimals behind the decimal point. Let's apply that change in the first next breaking release.

With {exponential:{lower:1e-100,upper:1e100} I just mean like set a value close to -Infinity and +Infinity, but you're right, this is not a real solution. It's a good idea to allow enteringlower: 0 (not -Infinity as I wrongly mentioned before) and upper: Infinity as lower and upper bound, we should implement a fix for that. Would you be interested in creating a PR with such a fix?

balagge commented 8 years ago

I would certainly be interested. I've never previously done a PR, so is there something special to know about it?

balagge commented 8 years ago

Summary of requirements

Requirement: math.format() shall enable conventional decimal formatting of numbers. It shall be guaranteed that no exponential notation and no loss of digits occur after the decimal point. Should work for any number, BigNumber and Complex values.

Such formatting will be requested either by

as options to math.format(value, options).

Details

Following changes are required:

  1. Extend math.format(value, {notation: "auto", exponential:{lower:low, ,upper:up}}) to accept low = 0 and/or up=Infinity for all kinds of numeric value types. (Currently behaves as intended for number and Complex, but throws exception for BigNumber value.)
  2. Same if notation: "auto" is not explicitly specified, but defaulted.
  3. Change the behavior of math.format(value, {notation:"fixed"}, if an explicit precision option is not given. (Currently rounds to integer for numbers, BigNumbers and Complexes as well. This behavior copies the behavior of javascript toFixed(). Instead, the BigNumber.toFixed() behavior is preferred.)
balagge commented 8 years ago

test javascript code

Note: javascript toFixed() as well as math.format() default behaviors are included for reference only.

/* jshint node: true */
'use strict';

var math = require('mathjs');

var numbers = [
1.1234567890123456789e+30, 
1.1234567890123456789e-30, 
-1.1234567890123456789e+30,
-1.1234567890123456789e-30, 
math.bignumber("0.12345678901234567890123456789012345678901234567890123456789012345678901234567890"), 
math.bignumber("12345678901234567890123456789012345678901234567890123456789012345678901234567890"),
math.complex({re:1.1234567890123456789e+30, im:1.1234567890123456789e-30}),
math.complex({re:-1.1234567890123456789e+30, im:-1.1234567890123456789e-30})
];

function tryFormat (n,f) {
    try {
        console.log(f.name, ":", f(n));
    } catch (e) {
        console.log(f.name, " returned error: ", e.toString());
    }
}

function toFixed (n) {
    return n.toFixed();
}

function formatDefault (n) {
    return math.format(n);
}

function formatLimitless (n) {
    return math.format(n, {exponential:{lower:0, upper:Infinity}});
}

function formatAutoLimitless (n) {
    return math.format(n, {notation:"auto", exponential:{lower:0, upper:Infinity}});
}

function formatFixedDefault (n) {
    return math.format(n, {notation:"fixed"});
}

var i,n;
for (i=0;i<numbers.length;i++) {
    n=numbers[i];
    console.log("\n"+n.toString(), "type: ", math.typeof(n));
    tryFormat(n, toFixed);
    tryFormat(n, formatDefault);
    tryFormat(n, formatLimitless);
    tryFormat(n, formatAutoLimitless);
    tryFormat(n, formatFixedDefault);
}
balagge commented 8 years ago

Current Output

** means a change is required !! means strange behavior of toFixed() (for reference only)

    1.1234567890123457e+30 type:  number
!!  toFixed : 1.1234567890123457e+30
    formatDefault : 1.1234567890123457e+30
    formatLimitless : 1123456789012345700000000000000
    formatAutoLimitless : 1123456789012345700000000000000
    formatFixedDefault : 1123456789012345700000000000000

    1.1234567890123457e-30 type:  number
!!  toFixed : 0
    formatDefault : 1.1234567890123457e-30
    formatLimitless : 0.0000000000000000000000000000011234567890123457
    formatAutoLimitless : 0.0000000000000000000000000000011234567890123457
**  formatFixedDefault : 0

    -1.1234567890123457e+30 type:  number
!!  toFixed : -1.1234567890123457e+30
    formatDefault : -1.1234567890123457e+30
    formatLimitless : -1123456789012345700000000000000
    formatAutoLimitless : -1123456789012345700000000000000
    formatFixedDefault : -1123456789012345700000000000000

    -1.1234567890123457e-30 type:  number
!!  toFixed : -0
    formatDefault : -1.1234567890123457e-30
    formatLimitless : -0.0000000000000000000000000000011234567890123457
    formatAutoLimitless : -0.0000000000000000000000000000011234567890123457
**  formatFixedDefault : -0

    0.1234567890123456789012345678901234567890123456789012345678901234567890123456789 type:  BigNumber
    toFixed : 0.1234567890123456789012345678901234567890123456789012345678901234567890123456789
    formatDefault : 0.1234567890123456789012345678901234567890123456789012345678901235
**  formatLimitless  returned error:  Error: [DecimalError] Invalid argument: toExpNeg: -Infinity
**  formatAutoLimitless  returned error:  Error: [DecimalError] Invalid argument: toExpNeg: -Infinity
**  formatFixedDefault : 0

    1.234567890123456789012345678901234567890123456789012345678901234567890123456789e+79 type:  BigNumber
    toFixed : 12345678901234567890123456789012345678901234567890123456789012345678901234567890
    formatDefault : 1.234567890123456789012345678901234567890123456789012345678901234567890123456789e+79
**  formatLimitless  returned error:  Error: [DecimalError] Invalid argument: toExpNeg: -Infinity
**  formatAutoLimitless  returned error:  Error: [DecimalError] Invalid argument: toExpNeg: -Infinity
    formatFixedDefault : 12345678901234567890123456789012345678901234567890123456789012345678901234567890

    1.1234567890123457e+30 + 1.1234567890123457e-30i type:  Complex
    toFixed  returned error:  TypeError: n.toFixed is not a function
    formatDefault : 1.1234567890123457e+30 + 1.1234567890123457e-30i
    formatLimitless : 1123456789012345700000000000000 + 0.0000000000000000000000000000011234567890123457i
    formatAutoLimitless : 1123456789012345700000000000000 + 0.0000000000000000000000000000011234567890123457i
**  formatFixedDefault : 1123456789012345700000000000000 + 0i

    -1.1234567890123457e+30 - 1.1234567890123457e-30i type:  Complex
    toFixed  returned error:  TypeError: n.toFixed is not a function
    formatDefault : -1.1234567890123457e+30 - 1.1234567890123457e-30i
    formatLimitless : -1123456789012345700000000000000 - 0.0000000000000000000000000000011234567890123457i
    formatAutoLimitless : -1123456789012345700000000000000 - 0.0000000000000000000000000000011234567890123457i
**  formatFixedDefault : -1123456789012345700000000000000 - 0i
josdejong commented 8 years ago

Thanks @ballage, your requirement description sounds correct.

Since this will be a breaking change in the behavior of math.format, we will have to apply the change to version 4, and I will collect all breaking changes in #682 and have created a v4 branch. And since it will be a breaking change anyway, we may want to reconsider whether upper and lower expects an absolute value (like 10e4) or an exponent (like 4), which may be more intuitive.

About creating a PR: you can clone the project, create a new branch to work on the issue: change the code, add/update corresponding unit tests, update docs. Then create a PR (to the v4 branch) so I can review and merge it.

balagge commented 8 years ago

I was away for a while, but now I am back and I will start working on this.

Yeah, maybe using exponents as limits is a bit more intuitive. I am still wondering a bit about the limits, though. Currently the lower bound is inclusive and the upper bound is exclusive. I am tempted to reconsider and may try making them both inclusive, so {lower:-3, upper:+3} would mean:

I know its kinda 'categorical imperative' and/or 'gut feeling' for an IT guy to use half-closed intervals whenever some kind of limits are needed. But frankly, I don't see the point here to use [lower, upper) kind of limits.

josdejong commented 8 years ago

Good point, makes sense to include both upper and lower limit. And I like the exponents as limits more too I think. It would also look more logic when defining them both as Infinite instead of having the lower limit be zero.

josdejong commented 6 years ago

In the v4 branch, I've now changed the default value of precision to undefined too when notation='fixed':

math.format(1234567890.1234, {notation: 'fixed'}) 
// returns '1234567890.1234 instead of '1234567890'

I've also changed lower and upper such that you have to pass exponents instead of values:

// mathjs v3.x
math.format(2000, {exponential: {lower: 1e-2, upper: 1e2}}); // returns '2e+3'

// mathjs v4.x
math.format(2000, {lowerExp: -2, upperExp: 2}); // returns '2e+3'
josdejong commented 6 years ago

mathjs v4 is released now, closing this issue.

mirabilos commented 5 years ago

I just needed this, and because it’s apparently impossible in py3k and JavaScript, I wrote a small shell script https://evolvis.org/plugins/scmgit/cgi-bin/gitweb.cgi?p=shellsnippets/shellsnippets.git;a=blob;f=mksh/fixfloat.sh;hb=HEAD to postprocess the numbers and dissolve the scientific notation using string operations.

zohaibtahir commented 3 years ago

From this code, you can convert 2.342*10^3 into 2342 https://github.com/zohaibtahir/Math-Calculations/blob/main/Scientific-notation-to-real_number.js

mirabilos commented 3 years ago

@zohaibtahir yours won’t work for bignums because it uses naïve floating point operations, with the loss of precision that entails.