search

RAIVNLab / supsup

Code for "Supermasks in Superposition"
117 stars 19 forks source link

About fixing the bias in the layers #22

Open b224618 opened 2 years ago

b224618 commented 2 years ago

Hi,

Thanks a lot for sharing the code. I have a small question. It seems that the bias of the layers are not masked by the score and not freezed either, and I'm confused about this. Is the bias term different from other trainable parameters (weights)?

Thanks!

mitchellnw commented 2 years ago

Hello,

Thanks for the question! Do you mean in the convs? We are always setting bias=False in the convs (e.g., https://github.com/RAIVNLab/supsup/blob/master/models/builder.py#L29-L66).

b224618 commented 2 years ago

Hi,

Thanks for the quick reply! Now I understand, I have misunderstood the code. By the way, I'm still little bit confused why can't the bias be masked in the same way as the weights?

mitchellnw commented 2 years ago

Yes, it could! I don't know if that would make much of a difference.

b224618 commented 2 years ago

Got it! Thanks a lot for your help!