search

Patashu / break_eternity.js

A Javascript numerical library to represent numbers as large as 10^^1e308 and as small as 10^-10^^1e308. Sequel to break_infinity.js, designed for incremental games.
MIT License
120 stars 43 forks source link

Unexpected NaN from sqrt() #164

Open hypcos opened 5 months ago

hypcos commented 5 months ago

I expected that X.sqrt() should give similar result to X.pow(0.5), but it doesn't. Especially when the layer is 1. Example:

> new Decimal(1e-16).pow(0.5)+''
< '1.000000000000002e-8'
> new Decimal(1e-16).sqrt()+''
< 'NaN'
> new Decimal(1e16).neg().pow(0.5)+''
< 'NaN'
> new Decimal(1e16).neg().sqrt()+''
< '99999999.99999979'
James103 commented 1 month ago

This is caused by incorrect handling of negative mag in sqrt: https://github.com/Patashu/break_eternity.js/blob/96901974c175cb28f66c7164a5a205cdda783872/break_eternity.js#L2157-L2158 When layer is 1 and mag is negative, the sqrt function tries to take the logarithm of a negative number, which is NaN.