Closed jmhessel closed 1 year ago
Hey!
The weights were pushed as float16
as this is what is used for inference
. We are going to add a line mentioning that the training was done un bfloat16
but there should not be issues with performances when training no. There is an issue with training in float16
as it was reported here #25065, which is expected.
Not by that default, if you use LlamaXXXX
the dtype will be torch's default float32
.
Hey @ArthurZucker ! Thanks for the reply :-) When you say "The weights were pushed as float16 as this is what is used for inference" --- is this what meta used in the llama2 paper for their results? I guess I am wondering why not also do bfloat16 for inference, particularly because of the potential fine-tuning issue when fine-tuning in float16 ? I could probably just do the conversion myself and host them on my huggingface, but just wondering the rationale, if possible
If you look at this or just try running a model you'll see that it is in fp16
. Main reason is that it's faster and should not really induce performance loss. But training was done in bf16
.
Aha! Gotcha :-) seems like if the official implementation does it this way, good to have it this way in huggingface. Maybe I'll do a bfloat16 conversion on my own and do some experiments, but probably not a big deal either way. Thanks!
FYI, for future readers, something related to this bfloat conversion was made: https://github.com/huggingface/transformers/commit/015f8e110d270a0ad42de4ae5b98198d69eb1964#diff-110a445233a8b15a0875998eeaf75cb8607b38a5daa736291dd058766879bbddL259-R273
It isn't clear to me what this change actually does, but it looks like the codellama weights are in bfloat16 https://huggingface.co/codellama/CodeLlama-7b-hf
not sure if this was intended but it might be worth trying the conversion for the original models in bfloat16 :-) (which I might try)
Once and for all, the dtype
of the checkpoints on the hub is only used if you set torch_dtype = "auto"
when you initialise the checkpoints. Otherwise, the torch_dtype
will be used to cast the checkpoints from the initialization type (so torch's float32
) to this torch_dtype
(only when you are using the auto
API.
The reason why we used the torch_dtype = torch.floa16
is because that the inference dtype, and thus for most common usages were you just want something to work out of the box, the type that should be used.
Following up on this, so what is the recommended dtype for llama2 inference? I assumed it's torch.float16
, given this thread and also I've always been working with the assumption that the dtype in config.json
is the recommendation. However, (1) I saw NaN issues inferencing with torch.float16
, which went away after switching to torch.bfloat16
; (2) the config for codellama specifies torch.bfloat16
, as Jack pointed out above.
@ArthurZucker Meta officially recommends bf16 though https://github.com/meta-llama/llama3?tab=readme-ov-file#access-to-hugging-face.
It is not obvious to me that casting bf16 -> fp16 -> bf16 does not lead to degraded perf.
It does lead to degraded perf. we pushed bfloat16 and the script respects that now
System Info
transformers
version: 4.31.0Who can help?
@ArthurZucker
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
Hi there!
credit to @dirkgr and @jacob-morrison for finding this LoC !
The facebook -> huggingface conversion script for llama/llama2 appears to cast weights to float16. While llama1 was distributed in fp16:
llama2 seems to be bfloat16
The casting differences are small in both absolute and percent terms (here's a random weight)
but, in theory, this could lead to a small performance degradation (e.g., https://github.com/facebookresearch/llama/issues/634).
Expected behavior
I think llama2 should probably be saved in bfloat16 rather than cast to float16 and then saved in huggingface format.