google / aqt

Apache License 2.0
248 stars 25 forks source link

Fix bug of FP numeircs. #710

Closed copybara-service[bot] closed 6 days ago

copybara-service[bot] commented 1 week ago

Fix bug of FP numeircs.

  1. When there is no exponent bits, e.g., e0mX, compute the maximum representable value by max subnormal instead of max normal.

  2. Deal with overflow: clip to max subnormal at the end of round when there are no exponent bits and subnormal presents in the numerics.

  3. Use 16-bit noise for deterministic SR tests.

  4. Use random bits to generate stochastic rounding noise in the subnormal case, instead of using jax.random.uniform.

  5. Cast test inputs to BF16.

  6. Add a test case to verity e1m2 is equivalent to e0m3.

  7. Add more tests to test_fp_round and test_fp_round_old_vs_new.