search

x448 / float16

float16 provides IEEE 754 half-precision format (binary16) with correct conversions to/from float32
MIT License
66 stars 8 forks source link

Added SmallestNonZero constant #46

Closed janpfeifer closed 6 months ago

janpfeifer commented 6 months ago
janpfeifer commented 6 months ago

Thanks for the review @fxamacker ! I added the test you suggested.

A reminder that I shouldn't blindly trust the AI that suggested the wrong smallest float16 ... Gemini suggested me that the exponent set to 1 was what would make the number be considered de-normalized (!?). Does this stand ?

In any case, I also tested that the NVidia GPU -- what I'm aiming at -- had the same interpretation of the number and it stands:

image (snapshot of Jupyter notebook cell, using gonb)

Btw, your library allowed me to add support for float16 when training ML models in GoMLX, which in turn almost doubled the speed of my GNN (Graph Neural Network) model training -- which I'm working on now. Many thanks for putting it together.

And on that note, let me add my interest on adding support for bfloat16 :smiley: -- the newer GPUs and Google's TPU support that.

x448 commented 6 months ago

@fxamacker thanks for reviewing. PTAL at #48 (it reduces linter noise spotted here).

fxamacker commented 6 months ago

Btw, your library allowed me to add support for float16 when training ML models in GoMLX, which in turn almost doubled the speed of my GNN (Graph Neural Network) model training -- which I'm working on now. Many thanks for putting it together.

Nice! :+1:

It was @x448 who wrote/contributed this float16 library to my CBOR codec and I'm happy it was extracted out from fxamacker/cbor as a standalone package for other projects like GoMLX can use it too! Thanks @x448!

And on that note, let me add my interest on adding support for bfloat16 😃 -- the newer GPUs and Google's TPU support that.

@x448 PTAL at bfloat16? :pray:

janpfeifer commented 6 months ago

Thanks for explanation @fxamacker ! And I was not aware of CBOR, very neat. Clearly, I was in Google for too long... (using protobuffers)

Btw I just retired from Google last Wednesday, maybe that's why the old scholar link no longer work. Thanks for the suggested alternative (although the page is incomplete)-- it seems I'll have to create my own page.

In Google it was really interesting indeed -- even though I didn't work on Gemini: I worked with YDF, SimpleML for Sheets, TF-GNN and other internal projects. The only drawback for me was that being a manager the last years I rarely had the time to actually code / experiment myself. One of the reasons I started investing my time creating my own OSS projects.