waylonflinn / weblas

GPU Powered BLAS for Browsers :gem:
MIT License
702 stars 43 forks source link

Unit Test Float Encode Exhaustively #11

Open waylonflinn opened 8 years ago

waylonflinn commented 8 years ago

The floating point encode step is a tricky bit to get right. Issue #10 shows that numerical stability issues can make a big difference in some edge cases.

Encoding each unique floating point value as an element of a 4098 square matrix should require only a few hundred of these matrices. This should be doable in a reasonable amount of time.

References

waylonflinn commented 8 years ago

Alternative Encode Methods

robertleeplummerjr commented 7 years ago

What is the current decimal precision? Is it better, for example than: https://github.com/gpujs/gpu.js/blob/develop/test/src/internal/precision.js#L7?