Open wizard1203 opened 1 month ago
Thanks for your issue @wizard1203, I'm not entirely sure how this relates to transformers
; this rather seems like an issue with torch
no?
Do you get similar issues with torch
parameters not from transformers
?
Do you get similar issues with
torch
parameters not fromtransformers
?
Did not try this. Because the lm_head is initialized by transformers
in the Auto Model and transformers
implement the parameter reuse, it is difficult to say which issue relates to.
An official averaging or parameter recursive function maybe safer for users who do not know many details.
System Info
transformers==4.32.1 torch=2.0.1
Who can help?
No response
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
In a recursive way of averaging model parameters using torch.distributed as dist.all_reduce(parameter, op=dist.ReduceOp.SUM) and then dividing the value by the number of workers, the result of the lm_head is actually the sum.
This bug may not only exist in the distributed allreduce, but also in other sum-divide averaging situations, like SWA algorithm which is frequently used in optimization, or the federated learning.
This bug is caused by lm_head parameters reusing https://discuss.huggingface.co/t/why-is-the-lm-head-layer-in-gpt2lmheadmodel-not-a-parameter/639.
Specifically, the lm_head reuses the parameters of the transformer.wte. During the recursive all_reduce, the
will sum both parameters of transformer.wte and the lm_head. However, when executing
of the transformer.wte, only the transformer.wte is averaged, because the number of references of avg_params[name] is at least two:
avg_params["transformer.wte.weight"]
andavg_params["lm_head.weight"]
. This results in the averaging error of lm_head, which actually is summation.One simplest way to avoid this problem is to use torch.distributed as dist.all_reduce(parameter, op=dist.ReduceOp.AVG). However, this limits many usage scenarios. Furthermore, this bug may cause other similar problems in the similar situation.
The minimal and fast reproducible example. Please use torch.distributed with 4 workers to launch it:
Expected behavior
Functions in the given examples
should have same outputs.