search

yule-BUAA / MergeLM

Codebase for Merging Language Models (ICML 2024)
745 stars 42 forks source link

Are the classification heads merged? #31

Closed SpeeeedLee closed 3 weeks ago

SpeeeedLee commented 1 month ago

In your code, in case of merging BERT, Roberta, etc., It seems that the classification heads were not merged, but separately keep and add back to the merged checkpoints for each task. Am I correct? Thanks!

yule-BUAA commented 1 month ago

Yes. Your understanding is right.

We do not merge the classification heads for BERT and Roberta since different tasks may have different numbers of classes.