search

steph1111 / PRECISE

Precision Rules Enforced Calculations In Scientific Environments
Other
0 stars 0 forks source link

sig_float: Incorrect sig figs for numbers with leading zeros and a decimal place #6

Closed steph1111 closed 1 year ago

steph1111 commented 1 year ago

0.0007 outputs as having 5 sig figs, it probably should have 1. I think????

steph1111 commented 1 year ago

that = sig_float("0.00007") that.sig_figs() 5

steph1111 commented 1 year ago 
- 7) 0.00033 -> 5

I thought this was fixed but guess not...

steph1111 commented 1 year ago 
+ 7) 0.00033 -> 2

test = sig_float("0.0004") test.sig_figs() 1 test.precision() 4 test.string '0.0004'