search

princeton-nlp / CoFiPruning

[ACL 2022] Structured Pruning Learns Compact and Accurate Models https://arxiv.org/abs/2204.00408
MIT License
188 stars 32 forks source link

prunable_model_size updated only in initialize_structured_head #54

Closed tatami-galaxy closed 9 months ago

tatami-galaxy commented 9 months ago

Hi this is a clarification question. In l0_module.py self.prunable_model_size is updated only in the initialize_structured_head function but not in either initialize_structured_mlp or in initialize_whole_mlp. Why is that? The mlp layers can also be pruned later I believe?

xiamengzhou commented 9 months ago

Hi! The prunable_model_size is updated in both initialize_structured_head and initialize_structured_mlp, as the parameters these two substructures govern overlap with the ones in initialize_whole_mlp and initialized_layer_structured_heads, they are not updated in the latter two functions.

tatami-galaxy commented 9 months ago

My apologies. I missed the line where prunable_model_size is being updated in initialize_structured_mlp. Thanks a lot for the quick response.