Open JiayiFeng opened 1 year ago
which model did you use?
I used a modified GPT-NeoX model, which is not officially supported. So I written a custom policy, and find this issue.
Maybe the
proj_layer = list(
map(lambda x: torch.cat([*x], dim=-1), zip(*proj_layer))
)
should be:
proj_layer = list(
map(lambda x: torch.cat([*x], dim=dim), zip(*proj_layer))
)
I guess.
okay so could you test it with other models?
When a attn_qkv
Layer
is set withn_fused>1
andreversed=False
, the shape of its sliced weight is incorrect.Seems that the root cause is here:
https://github.com/tunib-ai/parallelformers/blob/436573b05b9d47dba2234d60e874ff7bb57b725b/parallelformers/parallel/slicing.py#L79-L95
For a attn_qkv weight, the arg
dim
is 0. So when thereversed=False
andn_fused>1
, the tensor is chunked on the dim 0 and then concatenated on the dim 1. Which make its shape incorrect.