search

microsoft / knossos-ksc

Compiler with automatic differentiation
Other
45 stars 10 forks source link

Default Float to float #691

Open awf opened 3 years ago

awf commented 3 years ago

Cgen currently emits floats as doubles, but 99% of machine learning uses float32, so let's...

This might be a good time to typedef ks_FLoat etc in knossos.h.

awf commented 3 years ago

Note #358 is the bigger issue, this is just to change the default.

toelli-msft commented 3 years ago

Interesting. Generally I've always assumed defaulting to double was best, for reasons explained at Stack Exchange. Does 99% of ML use float32 because GPU memory is at a premium, and/or because it gives a regularisation benefit, and/or something else?

awf commented 3 years ago

The main reasons I can think of to use float are:

  1. You are storing large arrays of numbers and need to reduce your program's memory consumption.
  2. You are targeting a system that doesn't natively support double-precision floating point. Until recently, many graphics cards only supported single precision floating points. I'm sure there are plenty of low-power and embedded processors that have limited floating point support too.
  3. You are targeting hardware where single-precision is faster than double-precision, and your application makes heavy use of floating point arithmetic. On modern Intel CPUs I believe all floating point calculations are done in double precision, so you don't gain anything here.
  4. You are doing low-level optimization, for example using special CPU instructions that operate on multiple numbers at a time. So, basically, double is the way to go unless you have hardware limitations or unless analysis has shown that storing double precision numbers is contributing significantly to memory usage.

All of the above are true in machine learning on GPUs.

toelli-msft commented 3 years ago

Yes, that page is full of good explanations of why we would use float rather than double, if I'd read carefully enough!