I calculated the RMSE score on test data with model_performance(explain_test, score = "rmse")function from auditor, where explain_test is object explain fromDALEX for xgboost model and test set.
The score is equal to 0.16. Now, I calculate feature importance with model_parts with loss_function = loss_root_mean_square and n_sample = nrow(test). The RMSE score is equal 0.28.
I find a possible problem in code, look at https://github.com/ModelOriented/ingredients/blob/19a62decca73ad6c05bd7113fd091d20633ccbec/R/feature_importance.R#L195
so even if you give n_sample of the size of the full set, you do not get the whole set, but only the observations drawn with the return.
hbaniecki
commented
4 years ago
I calculated the RMSE score on test data with
model_performance(explain_test, score = "rmse")function fromauditor, whereexplain_testis objectexplainfromDALEXfor xgboost model and test set. The score is equal to 0.16. Now, I calculate feature importance withmodel_partswithloss_function = loss_root_mean_squareandn_sample = nrow(test). The RMSE score is equal 0.28. I find a possible problem in code, look at https://github.com/ModelOriented/ingredients/blob/19a62decca73ad6c05bd7113fd091d20633ccbec/R/feature_importance.R#L195 so even if you give n_sample of the size of the full set, you do not get the whole set, but only the observations drawn with the return.