Exercise 2: Functions with large gradients

[jknox13]

Are we expected to alter the algorithm in any way in order to account for functions with large gradients? For example, the function x^4 - 4x + 4 has a very large gradient for x < -3 and for x > 3, so the algorithm as written will only converge for very small sigma (order 10 ^ - 2 and smaller). So do we need to alter the algorithm to take functions like this into account, or am I just writing my algorithm for gradient_descent incorrectly?

[cswiercz]

This is a good observation. In the case when f has large magnitude gradients one of the problems with this version of gradient descent is that, unless sigma is chosen appropriately, the next iterate can be be far away from the current iterate. One of the solutions is to pick a better right sigma.

The version asked for in the homework is a simplified version of the actual algorithm where an optimal sigma is determined at each step. For the purposes of determining if your implementation is sufficient for this homework your code will be tested on "relatively tame" examples with appropriately chosen sigma.