Investigate and Fix '128' Rounding Error

waylonflinn / weblas

GPU Powered BLAS for Browsers :gem:

MIT License

702 stars 43 forks source link

Closed waylonflinn closed 8 years ago

waylonflinn commented 8 years ago

Weird precision loss for values near 128. Possibly related to floating point encode.

See #4 for test output.