search

THUDM / SwissArmyTransformer

SwissArmyTransformer is a flexible and powerful library to develop your own Transformer variants.
https://THUDM.github.io/SwissArmyTransformer
Apache License 2.0
978 stars 92 forks source link

MixtralMlpMixin()这个函数里面moe只是计算专家的logits但是没看到分发逻辑 #170

Open AlenjandroWang opened 8 months ago

AlenjandroWang commented 8 months ago

https://github.com/THUDM/SwissArmyTransformer/blob/main/sat/model/official/mixtral_model.py

1049451037 commented 8 months ago

在这里：

https://github.com/THUDM/SwissArmyTransformer/blob/eb4fac918cc86b304840872d4dccaaaf1b477e37/sat/transformer_defaults.py#L156-L202