add learning rate multiplier to network's graph

BVLC / caffe

Caffe: a fast open framework for deep learning.

http://caffe.berkeleyvision.org/

Other

34.09k stars 18.7k forks source link

add learning rate multiplier to network's graph #6244

Closed nic25 closed 6 years ago

nic25 commented 6 years ago

https://github.com/BVLC/caffe/blob/daf013931b31ed9c95250a89d09b7220badbcefe/python/caffe/draw.py#L89

IMHO including information about learning rate multipliers could be helpful when finetuning a network. We could:

add text information
change the color saturation to reflect the learning rate multiplier, either "washing out" layers where the LR is set to 0 or scaling the color saturation of each node depending of the learning rate

Noiredd commented 6 years ago

It might be useful, but in order to reduce clutter in the graph, I would recommend the lr_mult not be displayed by default - maybe it should be controlled via an optional argument?
As for the presentation, I don't think scaling the saturation is going to be a good idea, from the informational point of view. Text will work much better, hiding the label for layers with zero learning rate could also be considered.

I'd be glad to review a PR with this feature, if you wanted to implement it.

nic25 commented 6 years ago

Sure!

Noiredd commented 6 years ago

Excellent! I will close this issue for now - we shall have further conversation under the PR when you submit it.

nic25 commented 6 years ago

dysplaying LRM display lr dysplaying LRM/removing labels if LRM is 0 remove labels from layers with 0 lr

both work fine - i'll submit a pr when either can be optionally activated

Noiredd commented 6 years ago

@nic25 Could we retain the kernel_size/stride/pad/etc. info even when lr_mult is zero? Currently you're removing all information, while I think having the basic layer info regardless of the lr_mult would be useful. Other than that it looks good.