Compute MSE for LUNA, NLM, FF

[msbutler]

Creating this issue to document updates on computing MSE for the three models

[msbutler]

@vsavram @jscuds @mecunha @mcembalest you can now compute the log likelihood of luna and nlm objects after training. see the NLMdemo.ipynb for an example

object.get_log_l(x_train,y_train,x_valid,y_valid)

You can get the log likelhood of a feed forward object by calling

nn.objective_trace.min()

[msbutler]

One more thought here: given that an NLM (and LUNA) are two step algorithms (i.e. fit a NN, then fit a bayesian linear model), when evaluating the fit of a model there are two metrics to consider:

• feed_forward log likelihood (or mse, they are equivalent): evaluates how well the neural net performs. As Victor has demonstrated in optimizers analysis, nn.objective_trace provides the log likelihood. Given that this metric only evaluates the NN part of the model, it's insufficient to just use this when comparing the performance across NLMs.

• nlm test log likelihood: evaluates how well the whole nlm performs. See the NLMDemo.ipynb. This is the primary way to compare fit across models. See figure 4 in the luna paper for another example.

The last general performance metric to consider is epistemic uncertainty. Aside from creating nice plots, we're still trying to figure out how to calculate that (EDIT: i believe it's just standard deviation of posterior predictive samples within gap and outside gap). I don't think it's as important for the 205 paper though.