Razenpok / BreakInfinity.cs

Double replacement for numbers that go over 1e308
MIT License
212 stars 33 forks source link

Exponent might be wrong #20

Open Khaeops opened 10 months ago

Khaeops commented 10 months ago

While trying out this package I noticed that the exponents don't seem to be working as expected. Typing '1000' into the left field in the Unity Inspector yields a value of 1e6, and typing '1000000' (1 million) yields a value of 1e21. I would imagine the expected values should be 1e3 and 1e6, respectively.

asdfchlwnsgy1236 commented 2 months ago

That is due to the way the custom drawer for BigDouble works. Its value is updated every time the field content changes. For example, when you type in 1000 into the mantissa field, this is what happens:

  1. Type in 1.
  2. Field is 1.
  3. Value is {mantissa: 1, exponent: 0} and displayed as 1.
  4. Type in 0.
  5. Field is 10.
  6. Value is {mantissa: 10, exponent: 0}.
  7. Value is normalized to {mantissa: 1, exponent: 1} and displayed as 10.
  8. Type in 0.
  9. Field is 100.
  10. Value is {mantissa: 100, exponent: 1}.
  11. Value is normalized to {mantissa: 1, exponent: 3} and displayed as 1000.
  12. Type in 0.
  13. Field is 1000.
  14. Value is {mantissa: 1000, exponent 3}.
  15. Value is normalized to {mantissa: 1, exponent 6} and displayed as 1000000.

So, this is a problem in the custom drawer instead of BigDouble itself.