search

huggingface / hub-docs

Docs of the Hugging Face Hub
http://hf.co/docs/hub
Apache License 2.0
285 stars 245 forks source link

Wrong max number in in BF16 documentation #1425

Closed alankashkash closed 3 weeks ago

alankashkash commented 1 month ago

This repository is focused on the Hub experience and documentation. If you're facing an issue with a specific library, please open an issue in the corresponding GitHub repo. If you're facing an issue with a specific model or dataset, please open an issue in the corresponding HF repo.

Bug description. Incorrect number in a sentence "In fp16 the biggest number you can have is 65535 and any number above that will result in an overflow." at https://huggingface.co/docs/transformers/perf_train_gpu_one#bf16

Describe the expected behaviour A correct number should be 65 504, as given by other sources: https://en.wikipedia.org/wiki/Half-precision_floating-point_format https://blogs.mathworks.com/cleve/2017/05/08/half-precision-16-bit-floating-point-arithmetic/ https://blog.demofox.org/2017/11/21/floating-point-precision/

Additional context Add any other relevant context or screenshots here. Please share details such as browser when appropriate.

pcuenca commented 1 month ago

I agree:

>>> torch.finfo(torch.float16)
finfo(resolution=0.001, min=-65504, max=65504, eps=0.000976562, smallest_normal=6.10352e-05, tiny=6.10352e-05, dtype=float16)

Would you like to open a PR to fix, @alankashkash?

alankashkash commented 4 weeks ago

Sure, just submitted https://github.com/huggingface/transformers/pull/33658 It's my first contribution to HF, so thanks for the encouragement @pcuenca!