search

PyImageSearch / Deep-Learning-for-Computer-Vision-with-Python

2 stars 1 forks source link

A problem about learning rate finder #6

Closed thedrewyang closed 3 years ago

thedrewyang commented 3 years ago

Please use this template while reporting bugs inside DL4CV. Issues related to personal project questions or code that we haven't authored will automatically be closed.

Are you using one of our development environments?

We provide a preconfigured AWS AMI and a VM to help our customers skip the environment setup part and jump straight to development. We highly suggest using one of our development environments to run our code.

If the issue is not related to code, then you can simply skip this question.

If you are using your own development environment then please provide the following details.

Describe the issue

Hi, I'm wondering that regarding to this picture (https://pyimagesearch.com/wp-content/uploads/2019/08/learing_rate_finder_lr_plot.png) which is on August 5, 2019 's post. When lr=1e-3, the loss is on the way which is decreasing, on the other hand, when lr = 0.8, the loss is on the way which is increasing. But whether lr = 1e-3 and lr = 0.8 will perform as good as each other since their corresponding losses are the same which is around 1.75.

Or is there any better explanation for this learning rate finder? Because I can not find the answer to this question on dl4cv books, so I'm asking here.

Thanks in advance!

Best, Drew Yang

Any other information

ariG23498 commented 3 years ago

Hey @splash1123 This is a great question. According to your doubts, the loss is indeed the same at 1e-3 and 0.8, but that is not the main focus here. What we need to understand here is the slope of the curve. When the LR is close to 1e-3 the slope of loss is negative, that means that the changes in the weights are such that the model will converge (minimization of loss). On the other hand, when the LR is 0.8 the slope is positive, and the model diverges (maximization of loss).

The process of learning rate finder is more about the slope and understanding the pattern of LR change with loss. We need to observe the slope of the curve and take a better (guided) guess into finding a better LR.

Hope this helps you understand. 😄

thedrewyang commented 3 years ago

Hi @ariG23498, Thanks to your comprehensive explanation ! I got it now.😉