Closed scott-griffiths closed 2 months ago
At the moment the autoscaling algorithm uses
log2 = int(math.log2(max_float_value))
The int() rounds towards zero, but we want to find the largest power of two less than this max value. Therefore we should round to -inf and use math.floor.
int()
math.floor
Need to write some failing tests (perhaps compare with gfloat) and fix it.
At the moment the autoscaling algorithm uses
log2 = int(math.log2(max_float_value))The
int()rounds towards zero, but we want to find the largest power of two less than this max value. Therefore we should round to -inf and usemath.floor.Need to write some failing tests (perhaps compare with gfloat) and fix it.