search

uber / causalml

Uplift modeling and causal inference with machine learning algorithms
Other
4.9k stars 762 forks source link

Several questions about Meta-Learner #486

Open datasciyj opened 2 years ago

datasciyj commented 2 years ago

Hi, thanks for this awesome toolbox for uplift modeling. I have data for one treatment and one control group and used XGBoost as a base regressor. I'm trying to compare results from S, T, and X learner, and got some questions as below.

1) I made a pivot table for each model to see what average outcome values for control, test, and treatment effect(tau) by groups. S learner and X learner got very similar results for each average value which also make sense to me in terms of interpretation, but T learner resulted in very high value only for average test outcome of control group. I guess there are two things which could be related to this unexpected result. One is that the number of data assigned to treatment is only 4% of entire data. The other one is that I have a lot zeros for dependent variable so the distribution is highly right skewed. The number of zeros are 1000 times more than non-zeros. I was wondering what kind of factors can actually lead to this result and what that means exactly by high test outcome value for control group only.

2) Regarding the imbalance between treatment and control data, if that is a problem that needs to be handled, I'm curious if there is any recommendation there such as upsampling.

3) I found I can't have NaNs for Xlearner to let the model calculate propensity scores. Thus, I imputed Null values using simple imputer for test. I was also curious if there is any recommendation in terms of imputation for propensity score calculation.

Thanks in advance.

t-tte commented 1 year ago

Regarding the first two questions, it is known that the T-learner suffers when treatment and control classes are imbalanced. In fact, the X-learner is largely motivated by this observation. In your case, I would simply use the X-learner.

Regarding the third question, it totally depends on the specifics of your situation. Are you analysing a randomised experiment? If so, a common trick is to set the propensity score at 0.5. If you're analysing a non-randomised treatment, the first thing you should do is to use your subject matter expertise to only use those variables in the propensity score calculation that you have a reason to believe are actually related with selection to treatment (that is, don't throw in hundreds or thousands of variables at random). If you still have missing values after the previous step, refer to e.g. this paper to decide what to do next.