Training of predictors based on MOE model

[mailonghua]

Hello everyone, I want to train the FFN low-rank predictor for the MOE model,so how should i choose the point to collect training data? Switch Transformers