jsoftware / jsource

J engine source mirror
Other
645 stars 91 forks source link

(8!:2) formatting error on extended precision argument #136

Open rhodgkinson opened 2 years ago

rhodgkinson commented 2 years ago

Brian McGuinness reported different results for extended precision formatting using (8!:2) as follows: !100x 93326215443944152681699238856266700490715968264381621468592963895217599993229915608941463976156518286253697920827223758251185210916864000000000000000000000000 'c0' (8!:2) !100x 93,326,215,443,944,163,159,976,979,467,798,536,412,765,149,252,739,980,661,474,384,325,625,785,282,152,304,831,674,467,853,997,780,872,771,564,755,772,036,613,628,485,669,284,340,068,195,836,863,115,824,201,728

Rob Hodgkinson verified with these examples, suggesting a loss of precision around "16 significant figures"... !22x NB. OK ... 1124000727777607680000 'c0' (8!:2) !22x 1,124,000,727,777,607,680,000

!23x NB. Loss of precision around 16 significant figures 25852016738884976640000 'c0' (8!:2) !23x 25,852,016,738,884,978,212,864

!30x NB. and again... 265252859812191058636308480000000 'c0' (8!:2) !30x 265,252,859,812,191,032,188,804,700,045,312