Floating point incrementation erratic behaviour

[mattst]

Using: Sublime Text 3 Build 3059, Linux Mint 13 LTS. Using: InsertNums v. 2.0.1 (version from: Insert Nums/package-metadata.json)

When using InsertNums to insert a number sequence with a floating point step value there are some very strange results.

// Example of `1:.1` with 10 selected regions.

1.0
1.1
1.2000000000000002
1.3000000000000003
1.4000000000000004
1.5000000000000004
1.6000000000000005
1.7000000000000006
1.8000000000000007
1.9000000000000008

I've done some screen captures in order to show a variety of examples.

Zip file with all images: sockshare.com - no waiting

Hope this helps.

[jbrooksuk]

Interesting. The same occurs on OSX 10.9 too.

@FichteFoll any ideas?

[FichteFoll]

Windows7, ST2220:

>>> .1
0.10000000000000001

Windows7, ST3062:

>>> .1
0.1

It's a precision issue with floating point numbers. Not 100% sure whether we should handle this, but if we were to, I'd probably specify some default maximum precision argument in format(), because the data type itself is unprecise and we need to fix it when converting into a string.

It also seems that the exact number of max printed significant numbers is Python version- and OS-dependant since it works for me on ST3.

@mattst, thanks for the great images btw, but you wouldn't have needed this many. ;)

[FichteFoll]

https://docs.python.org/3/tutorial/floatingpoint.html

It seems we have 2 possible solutions:

1. Set the default precision in a format call to something like 12 (16-18 significant digits are usually the critical zone). Since floating point number precision is limited to the number of significant digits and not precision, we'd have to adjust our "rounding" to that, somehow.
2. Use the decimal module. That could work for the built-in calculations but custom expressions with literal numbers will taint our precise decimal.Decimal("0.1") and similar, and make it unpredictable.

[jbrooksuk]

I think that option 1 sounds like the best solution, without having to introduce more changes.

[mattst]

@FichteFoll

"@mattst, thanks for the great images btw, but you wouldn't have needed this many. ;)"

I got a bit carried away. :) I was trying to show the variations in output, some work fine in my examples, as well as the format string workaround.

In your solution considerations you might consider this point:

The casual user (most users?) probably wants the precision only of the level they used as input. e.g. 1:.1~.1f, 1:.123~.3f, 1:.00001~.5f. Why not set the precision on that basis and those that want more precision can delve into the formatting options?

P.S. @FichteFoll - You realize that this erroneous ('oh, the shame') SublimeText issue was me as well? "In mousemap 4th mouse button is not referenced by "button4" instead its "button8". I only realized because you closed that issue 26 mins ago, so I've got back to back FichteFoll emails in my inbox about 2 different issues.

[FichteFoll]

@jbrooksuk Yes, definitely.

@mattst Thinking about that, I suppose you are right that there is in fact no actual use case where you'd need a higher precision for printing a number than the maximum precision of the two summands (since we're multiplying the step with whole numbers they are actually sums). However, there are two more things to consider:

1. If you specify either of the summands in scientific notation we can not accurately determine the desired precision from its string.

   >>> 1+1e-16
   1.0
   >>> 1+1e-15
   1.000000000000001

2. If the user enters a number like formatstring like 1:1.00000000000000000000000001, the result will not be as expected, since floats just are not that precise.

   >>> sf = "1.00000000000000000000000001"
   >>> "{num:.{prec}f}".format(num=float(sf), prec=len(sf) - 2)
   '1.00000000000000000000000000'

I generally prefer the "limit the significant digit number and round appropriately" approach but I'll have to see how well I can translate that into code. On my ToDo (but not high priority since it's a rather rare issue).

PS: No, I did not. I usually memorize avatars instead of names for users because they are easier to remember (brain stuff) and only then associate a name to an avatar. However, since you're using a generated gravatar I just ignored that because they look too similar.