search

stanfordmlgroup / ngboost

Natural Gradient Boosting for Probabilistic Prediction
Apache License 2.0
1.64k stars 215 forks source link

Potential bug in score method for sklearn API #218

Closed vdrao closed 3 years ago

vdrao commented 3 years ago

Hello,

Thank you for NGBoost! I think there may be a bug in the sklearn score API in the main NGBoost class (lines 317-318):

    def score(self, X, Y):  # for sklearn
        return self.Manifold(self.pred_dist(X)._params).total_score(Y)

Sklearn hyperparameter tuning functions like RandomizedSearchCV expect that better models are assigned higher (more positive) scores. However, as far as I can see, the total_score method above actually computes the total log loss, which means that smaller values correspond to better models. Should the score instead be the negative log loss?

alejandroschuler commented 3 years ago

Hi @vdrao, the method you mention actually calculates the negative log likelihood (when using the LogScore scoring rule). The correct sign is enforced in the definition of the score method for each distribution, e.g. for the normal.