Store convergence information during fitting

[cxrodgers]

This PR adds the following variables to class GLM. They are updated during the fitting process. I think they are useful for evaluating the performance of the fit afterwards (and even in the middle of fitting, if some error happens).

    `self.niter` : Number of iterations so far.
    `self.converged` : boolean, whether the convergence criterion has been reached
    `self.loss_by_iter` : list, the loss after each iteration
    `self.loss` : the current loss

[jasmainak]

hmm ... there is a callback argument in the init. What does this give that the callback does not? You might want to check out how we do it in the tests: https://github.com/glm-tools/pyglmnet/blob/master/tests/test_pyglmnet.py#L188-L195

Perhaps something to expose if users are interested in this, but I'm not sure if the end user should be fiddling with things like convergence.

[ghost]

As written, self.loss_by_iter and self.loss aren't actually storing the loss -- they're storing the info about step updates, but won't match up with _loss, _L2loss, or the loss/gradient cheatsheet. Also, niter will get reset to zero every time fit is run (which doesn't reset beta0_ or beta_).

That said, I'm an end user who would very much like to have the mainline package record convergence information in the fitted models, especially in light of convergence-related bugs like #275.

[jasmainak]

I'm fine exposing a callback function which the user can import and gives you the loss and gradient. Note that loss is not computed by default because it's expensive. You should also update the documentation somewhere to show this feature.

[cxrodgers]

Thanks both! I understand better now. I wasn't aware of the callback's existence. Looks like the thing I'm storing isn't actually the loss, oops. But, I note that it is the value that is used to define convergence, so it's useful to know.

I agree with @peterfoley605 , I find it very useful to be able to record convergence-related information. A major one is whether the model actually achieves convergence, which I don't know if it's currently possible to determine easily. If it didn't converge, the user knows to increase max_iter. This is especially important for comparing the results of different regularizations, e.g. in a cross-validation scheme.

[jasmainak]

cool! Let me know once you've updated the PR and I'll take another look

[jasmainak]

closed by #316