Plan for handling missingness

[emilycantrell]

Proposal to leave missing values as missing (at least for now)

I'm working on code for XGboost and Catboost models, and am deciding how to handle missingness. I am guessing Catboost will be the primary model type we use, so that's what I'm thinking about most right now, although of course our model choice may change depending how the models perform.

After reading about missingness in tree-based algorithms and thinking about the types of missingness in the CBS data, I'm inclined to leave the missing values as missing, with no imputation or other special handling. Here are notes about what led me to this plan:

• In the CBS data, missingness is generally due to structural missingness, not due to incomplete data collection*. For example, I am going to merge in data about a person's partner. Some people don't have partners, so all the partner data columns will be missing. It wouldn't make much sense to impute missing values for the partner's data, since these people don't have partners.
• For continuous variables, Catboost and XGBoost will automatically handle missing values. Catboost sets missing values to be lower than the minimum or higher than the maximum so that the trees can separate missing from non-missing values. Since most of our missingness is structural, I think this strategy of treating people with missingness as a distinct group makes sense.
• My impression from poking around the internet is that it's common to not impute missing values for Catboost.
• My instinct is that other data prep, like merging in partner data and feature engineering the "tab" files (files that list events over time), will make much more difference than our choice about how to handle missingness. I think other data prep and feature engineering should be higher priorities, and then we can test different ways of handling missingness if time permits.

*Caveat: This is my impression from other CBS files I've used, but I haven't seen the real data files for PreFer yet. If the missingness looks different than I expect, we can re-evaluate.

Question for the team

Does anyone have feedback on this proposal?

Blog posts

Here are a couple of relevant blog posts that test the impact of imputation in tree-based algorithms. Of course, results could be different with a different tree-based algorithm, different data, or different imputation method, but these articles might still be useful to consider:

• Against/neutral toward imputation: Your Dataset Has Missing Values Do Nothing! by Samuele Mazzanti _ Towards Data Science.pdf
• In favor of KNN imputation instead of XGBoost's default methods: https://towardsdatascience.com/xgboost-is-not-black-magic-56ca013144b4

[msalganik]

Thanks for the careful write-up @emilycantrell. After reading more about it, I agree. It seems like XGBoost and Catboost deal with missing data in a sensible way. Fruit on a plate!

P.S. If we try a regularized regression, then we are going to need to reconsider.