sgd - optimal learning rate with line search

[erodola]

Let me clarify the confusion on the orthogonal directions obtained with the optimal alpha in slide 66/127 of the lecture "Stochastic gradient descent".

Please read this while looking at the slide.

• Let's say we are at the point x_t (first red dot in the figure).
• The optimal alpha minimizes (globally) the function f along the direction -grad(f(x_t)).
• Let's call the optimal point y (second red dot).
• Since y is a minimum, the directional derivative of f(y) along direction -grad(f(x_t)) is equal to zero.
• By definition, this directional derivative can be written as the inner product between grad(f(y)) and the direction -grad(f(x_t)).
• Having this inner product equal to zero means that grad(f(y)) and -grad(f(x_t)) must be orthogonal.
• Since grad(f(y)) is also the direction of the next iteration of gradient descent, we get that successive directions of gradient descent with line search are always orthogonal.

[ShakirKhurshid]

y is minimum wrt what? isn't the point in the centre of the function (blue area) the minimum of the function and what is the direction of grad(f(y)) ?? can you please explain point 4, if you could draw out the directions? It would be really helpful.

[erodola]

I made the attached image to try clarifying the points you raised. Can you check if this works for you now? Please let me know if you have additional questions.

This image covers the first 4 points of the explanation above. Therefore you should be able to follow from point 5 on.