Could you show more details of the layer decay schedule when training the student for downstream tasks?

[LiYuhangUSTC]

A layer decay schedule is mentioned in Section 5.1 Implementation details "To avoid deteriorating the general representations obtained from the previous stage, a layer decay schedule is adopted to train the student model for all downstream tasks."

Could you show more details of the layer decay schedule? Or point me to the code/reference of the schedule?

Thanks

[Vickeyhw]

The layer decay schedule is implemented here:

https://github.com/pengzhiliang/G2SD/blob/master/G2SD/util/lr_decay.py#L6

[LiYuhangUSTC]

Thanks for the quick response!