search

danczs / NetworkAdjustment

13 stars 2 forks source link

How to select the adjust layer num K? #3

Open jqliu42 opened 3 years ago

jqliu42 commented 3 years ago

Hi,I'd like to ask a question about the hyperparameter K. How to select a suitable K for models with different depth? Do you use the same k for resnet20 and resnet56 ? Thanks for your help!

danczs commented 3 years ago

Thanks for your question. We use different k for networks with different detphs. For a network with L searched layers, we use k = L/3 , by which the number of the increased, decreased and unchannged layers is all L/3. By the way, if you wants to get better results, it is suggested to tune k in the range of L/6~L/2 and use 'update_num_decay' to reudce k during search.