loss increases sometimes

[dswah]

on some real problems, we see that the negative log likelihood (with l1 penalty) increases. the increase happens as a result of the update to theta.

here is a plot of the difference in loss after the update to theta. it looks like sometimes the loss increases!

but the lambda updates are protected by the backtracking:

[dswah]

the loss should NEVER increase due to a Theta update because the objective is convex and we have a closed-form solution for the update.