mu parametrization for gated-mlp and group-query attention

microsoft / mup

maximal update parametrization (µP)

https://arxiv.org/abs/2203.03466

MIT License

1.24k stars 88 forks source link

Open ftgreat opened 3 months ago

ftgreat commented 3 months ago

could you please give me advice about mu parametrization for gated-mlp and group-query attention, thanks very much.

@thegregyang @edwardjhu