pengzhiliang / G2SD

81 stars 3 forks source link

Could you show more details of the layer decay schedule when training the student for downstream tasks? #3

Open LiYuhangUSTC opened 1 year ago

LiYuhangUSTC commented 1 year ago

A layer decay schedule is mentioned in Section 5.1 Implementation details "To avoid deteriorating the general representations obtained from the previous stage, a layer decay schedule is adopted to train the student model for all downstream tasks."

Could you show more details of the layer decay schedule? Or point me to the code/reference of the schedule?

Thanks

Vickeyhw commented 1 year ago

The layer decay schedule is implemented here:

https://github.com/pengzhiliang/G2SD/blob/master/G2SD/util/lr_decay.py#L6

LiYuhangUSTC commented 1 year ago

Thanks for the quick response!