search

google-research / morph-net

Fast & Simple Resource-Constrained Learning of Deep Network Structure
Apache License 2.0
1.02k stars 149 forks source link

Multiple alive files output #189

Open ahndroo opened 4 years ago

ahndroo commented 4 years ago

What is the significance when I have multiple alive files in each morphnet_log folder? The are all very different, is there a method to choosing the 'best' one?

pkch commented 4 years ago

Alive files are generated throughout the training, at each step when summaries are computed. Typically, at the very beginning of the training, very few channels are pruned yet, so the alive files from early steps correspond to a very light pruning, i.e., a very small reduction in FLOPS. As training continues, the pruning becomes more aggressive. Eventually, the pruning stabilizes at some level. All of this can be seen in the FLOPS chart on Tensorboard.

The choice of the alive file can be based on the target FLOPS (or other complexity measure) of the final architecture. It is quite reasonable to try a few different point. For example one alive file can be from the end of the training when the FLOPS already reached their lowest level. Another one can be from the middle of the training where the FLOPS reduction is much more modest. Ultimately, retraining will show what quality can be reached with each structure.