JudasDie
commented
3 years ago Hello, thanks for sharing the great work. I'd like to ask some questions about how to training network if I add small sub-networks to an existing model (e. g. OCEAN). Can you give me some advice on learning rate setting (e.g., if warm up)? Is it a good choice to freeze the original model and only unfix the added subnetwork at the beginning of the training? And do I need to train both original part and added subnetwork together after then? Besides, how do you quickly verify that if one idea is valid? I’d appreciate it if you share any experience or lectures!
- For generality, you should train all modules end to end. If you can not (e.g. not convergence)，try to fix the original modules. However, this would limit the model capacity.
- First, try to use the same learning rate as the baseline model. Then, try its 0.1x and 10x version.
penghouwen
Hello, thanks for sharing the great work. I'd like to ask some questions about how to training network if I add small sub-networks to an existing model (e. g. OCEAN). Can you give me some advice on learning rate setting (e.g., if warm up)? Is it a good choice to freeze the original model and only unfix the added subnetwork at the beginning of the training? And do I need to train both original part and added subnetwork together after then? Besides, how do you quickly verify that if one idea is valid? I’d appreciate it if you share any experience or lectures!